Streaming API - Quick Intro

The Streaming API can be used to expose a near real-time stream of data from the Salesforce org in a secure and scalable way. The Streaming API utilizes a publish/subscribe model where we can create one or more named topics, each of which is associated with a SOQL query. External applications can then subscribe to one or more of these topics, using the Bayeux protocol. As and when the relevant data is created or updated, the published query is re-evaluated, and in case there is a change that alters the results of the query, a notification is published to all the applications that have subscribed to the topic.

Why Streaming API ?

This is an obvious question. Why Streaming API, when we already have REST and SOAP API. Well, the answer is simple - Callout Limits and overhead due to continuous polling to external applications. Apex Callouts and Outbound Messaging are the most preferred ways to integrate salesforce with external system. But, there will be times when one reaches their Callout Limits or want to use Outbound Messaging with non-SOAP endpoints.

If your application is currently polling the SOAP or REST API to detect changes, do consider using the Streaming API; and it will lead to better performance with lower resource consumption.

Limitations - Message Reliability

From the docs

Streaming API doesn’t guarantee durability and reliable delivery of notifications. Streaming servers don’t maintain any client state and don’t keep track of what’s delivered. The client may not receive messages for a variety of reasons, including:

  1. When a client first subscribes or reconnects, it doesn’t receive messages that were processed while it wasn’t subscribed to the channel.
  2. If a client disconnects and starts a new handshake, it may be working with a different application server, so it receives only new messages from that point on.
  3. Some events may be dropped if the system is being heavily used.
  4. If an application server is stopped, all the messages being processed but not yet sent are lost. Any clients connected to that application server are disconnected. To receive notifications, the client must reconnect and subscribe to the topic channel.

Live Demo

To get a glimpse of what exactly we are trying to build here you can watch the live demo of the integration by following the below steps:

  1. Open this link which is a Salesforce site I have created to let the guest users create Account in my org without any credentials.
  2. Next open this link (Node.JS app on Heroku) side-by-side the above window.
  3. Now create an account in the first window and see the same appearing in the second window in real-time (magic!).
Live Demo

What is Nodejs?

Watch this superb video by Pedro to understand Nodejs; its key features and what one shouldn't expect from Nodejs.

The Code


Create a Connected App

  1. Login to your Salesforce Org and in Setup go to Apps.
  2. Scroll down and under Connected app section Click New to create a connected app.
New Connected App in Salesforce
  1. Enter the details as shown below replacing the values relevant to your org
Connected App Configuration
Caution: OAuth Scopes is selected as Full Access(full) for demo purpose. Make sure you completely understand the repercussion before applying one
  1. Save and now you should be available with the Consumer Key and Consumer Secret which will be required in Node.JS configuration to connect to Salesforce.
Consumer Key and Secret

Now create a Push topic by executing the below code in the Developer Console. Alternatively you can create the same in Workbench.

 PushTopic pTopic = new PushTopic();
 pTopic.apiversion = 24.0;  
 pTopic.Name = 'AccountsTopic';
 pTopic.Description = 'New Account records';  
 pTopic.query = 'SELECT Id, Name FROM Account';  
 insert pTopic;  
 System.debug('Push Topic created with ID: '+ pTopic.Id);

Check the below line in log to confirm the Topic creation

Create Push Topic

Node.JS Code

  1. Download the zip or clone the github repository here.
  2. Run npm install on the prompt to install all the dependencies defined in package.json
npm install
  1. Once all the dependencies are installed successfully open the config.js file and replace the values for authentication fields as mentioned in the comments
Authentication Secrets
In case you are not available with security token for your org then you can obtain the same in your My Personal Settings : Reset My Security Token. But take caution if you are already using the token somewhere else.
Reser Security Token

Run Node.JS

  1. Go to the folder where you have the Node code and run node app.js in a prompt in that location. If everything is properly configured then you will see the below message.
node app.js
  1. Open the URL: http://localhost:3001 in a browser. Create an account in your org and you could see the same reflecting in real time in the browser (Thanks to Socket.IO).

Demo Video

Watch the video below to see the demo.

You've successfully subscribed to inteygrate
Welcome back! You've successfully signed in.
Great! You've successfully signed up.
Success! Your account is fully activated, you now have access to all content.