foreign [Music] thank you [Music] hey everyone my name is pranab and I am a senior developer advocate here at servicenow and I bring you another episode of creative toolbox on the topic stream connect so today I have a very special guest with me his name is Ander so I'd like to introduce him on the show hi Anders how are you doing hi kind of I'm very good thank you thanks for having me on perfect so Anders will be taking us through what what is stream connect and service now and personally speaking I am really keen on knowing about shame connect because when I came from the development background I was always confused with the stream connect spoke and a steam connect plugin I I was like what exactly is the difference so Anders will be taking us through that and for those of you who don't know about Creator toolbox created toolbox is a place where we showcase new features either ourselves or we have guests like handles who will be coming up and showing us some of the features so we have lot of other episodes available on our YouTube channel so you definitely should check our YouTube channel and also like we are doing this Utah content Series so there is like if you want to view another content you should go to Devlin dot Essence tutor you can see all the content related to Utah so without further Ado I'll just ask Anders to take us through simconnect and the stage is all yours excellent thank you very much oh it looks like Anders just got dropped I'm gonna give it another second corinthus to join in share your screen and let's jump into stream connect let me do that let me share my screen again there we go yes so did you lose me straight away you never heard anything I said right nope we didn't heard anything so okay yeah okay I was talking away to myself I was yeah anyway let's start again so hello everyone thanks for joining us today sorry about this technical little hiccup here so my name is Andrew Sanderson I work as a product manager in the automation engine team at servicenow and today we're going to look at how we can integrate service now with Kafka in two different ways so before we jump into what we can do I just want to spend a minute on what coca actually is so Kafka is an event streaming platform that allows you to read and write streams of events and it's available as a Cloud solution or an on-premise solution and Kafka is just really really good at handing large number of messages and shifts in between different systems so other systems can then produce data into this Kafka environment or read from it or both I guess so for service now traffic I can act as a buffer to store data that you push and pull between external systems and the service now platform and it generally simplifies the Integrations we have okay so Kafka is available in the service so stream connect is available in the search net utility release and we have a number of customers looking to various use cases they want to use so the most common ones being populate the cmdb and generate and track tasks in servicenow other use cases will be around Integrations between the servers now instance and any external systems and a lot of customers have already built Integrations between Kafka and the servicenow table apis for example but now stream connect can be used as a robust and scalable solution here foreign not just going to talk about stream connect we're also going to go back in time to the sort of San Diego release when we released a Spoke uh called the conflict Kafka rest proxy and it's available in the servicenow store so with the complement Kafka responses spoke there is also an application uh called the search net Kafka consumer which you can use to read or consume data from Kafka and write it to serve as that table used in the robust I think Anders is facing some technical issue let me bring him again just to see where it is just okay we were actually at the very interesting slide like about the difference between stream connect plugin and SLS spoke each other sometimes this thing happens sorry about it folks foreign [Laughter] having technical problems here today right no problem uh you can just share the screen again can you remember which slide I was on yeah you were on the slide on when you were explaining About Steam connect uh plugin versus the confluent one yeah this one here right yep cool right then the white keeps dropping out of it try again okay yes all good okay I'm sorry Okay so um so what I was talking about here right is that we have two different ways uh to connect to Kafka so if we go back in time to the search they have San Diego release we released a Spoke called the complement rest proxy on the servicenow store and we released an application called servicenow Kafka consumer which is using the spoke to read data and write data um between Kafka and servicenow um so not too long after we released this book we started working on connecting Kafka to substantly Kafka directly without using response so we wrote that Kafka clusters in our data centers across the globe and the servicenow instance with read and write data from those servicenow Kafka clusters so then you can configure replicator to synchronize the data between service net Kafka and your local Kafka so if we look at an overview of what the architecture could look like so for this spoke um you would configure your environment using flow designer you trigger an action and to either produce or consume data and the action will then connect to the conflictress proxy so if the if the rest proxy has a public IP then you can connect directly from the instance of course or you can also use the mid server to sort of pick up the message and do the communication to the rest proxy for you and then the rest proxy will either connect to the component Kafka on premise or the content Kafka or in the cloud of course and with stream connect solution we we don't use the spoke there are apis to read and write data from the service now software cluster which we call the Hermes messaging system customers can you ask them Point their replicator on their side directly to the service now Kafka then right or you can use Java code or any Kafka connectors I guess that's available out there to then get information in and out of the servicenow Kafka okay so if we go and have a little look at the conflict Kafka rest proxy spoke so what features do we have so it comes as a spoke it has a number of actions and the two I sort of picked for this presentation was to get messages so sort of read messages from Kafka and um so this is an example of what it would look like in a in a flow um the the rest proxy spoke is using the connection and credential Alias and you can use various protocols on Json Abram protograph of course and similar very similar on the producing side so approaching data from servicenow to Kafka um again it's a action called published messages and then you configure your topic and the format of the protocol you want to write this one then switching to stream connect so we're jumping back and forth here as you can tell um what features do we have here so with stream connect you we have access to the flow trigger so the flow trigger you basically again you decide what topic you want to um Listen to I should say right or sort of get messages from so every time there are messages in the in the in the topic or in the queue there then we will be able to read those messages and then process them with the logic of a flow so then sort of a low code option to to do this and when it comes to producing using stream connect then there is a producer step so you can build your own action so you can sort of build your simple payload with pills and drag drag and drop the pills in there and again produce to a topic there's also a scriptable API where you can as you can see on the screen here right you basically build your payload as a message and use the sort of send function there right to or method to to send information down there and we have a number of additional consumers as well so for for the scenarios where you want to write data to a table so you might already have your robust transform engine configured or your transfer Maps configured for this data uh you can reuse those um sort of transfer maps and and robust transfer maps to sort of switch them over to Kafka and then use those consumer Transformations using the consumers and there's also a script consumer that for examples where you might have a payload that might not be structured if you need to use some logic to get that that data parse or you want to do something else sort of enrich the data in some other way you can use the script consumer as well okay so we also have um ways of sort of looking at what topics you have available so you can sort of see the topics and for instances where there are domain separations and you can then root or sort of Link certain topics to certain domains and you do that with the sort of namespaces which provides a prefix so if the topic starts with a certain prefix you can route it to a certain domain Etc okay and looking at the consumers as well we can we can sort of see statistics and throughput for these consumers and so you can keep track of of the performance and how much lag it might be as you generate the data Etc into service now okay I think we are ready for uh some Hands-On demos here I was gonna say but it's going to be live demos so as always with live demos they could uh let's go see let's go switch to the right screen I'm going to jump around a little bit so bear with me before I okay so let's see if I'm gonna make this screen a little smaller there I'm going to bring in my terminal all right but now that it's the window it's a pretty good as well so here we go three windows so what are we looking at here this is the confluent environment that I've connected to um uh but it could be any coffee environment they just use conflict because they happen to have a an account there and I have a I have a cluster configured I have a number of topics configured um so I can see the sort of topics here and the messages that are being produced Etc into this uh into this environments so um and if I jump to my terminal again here it's a bit big but um that's going to see so I'm logged into I'm not logged in I'm going to log in to my rest proxy here um let's see is going to start the rest proxy so I have a little sheet script there to start it out okay sorry so this is the log file so in the log probe you can see the connections um as they happen as well right but all we're going to do here is from the servicenow instance I have a set of demo data here I called it demo sources we're not this is just a table with thousand records in it so that's what we're going to use and I also have another table which is empty at the moment and this is what we're going to write a message to then right so that's what we're going to do using Cloud assignment so I have a little flow here um you can see what I'm doing um it's just I just going to manually trigger this flow I'm not going to do anything Advanced here right but you can see it it looks at that demo Source table it takes 100 after 100 records um that should be enough for this summer and then we are basically creating a payload that we published using the the rest proxy message here right so um let me ask trigger this so as the flow is is triggering now the action would sort of take that payload push it to um the rest proxy as you can see happening there and you can see it here if my screen was big enough you can also see sort of messages coming in here to conflict right so that is very quickly how the best proxy how you would push messages from servicenow then right into um complement cloud in this case right but it could be any environment so it's finished here you can go and look at the execution details but for them for the sake of time I'm going to jump forward in My Demo here so what I can do now is it's going to open another flow and we're going to read some messages into this one so this is another topic just when my flow is right okay there we go so very similar here I have a flow that is actually runs every minute and it will go through and check for messages in that topic and then you do some magic in here of parsing the information as well then right then and write it to that table so I was going to activate this one and again not that we're looking at log pass a lot but you can see data coming through in that rest proxy as well one is it's um activates and goes and checks for messages wait for this to activate so and there's a quick question for you like right now you're sending data to the conflict right from service not distance that was yeah that was the first demo we did yeah exactly and then I now activated this flow and this flow is just waiting for data to come into service now so we have this table in the demo orders that I showed briefly before it's it's empty at the moment and if I go to my conflict here and produce a message I'm just going to take the standard message they provide here I'm going to click produce if it allows me to produce okay your message being processed and uh for the sake I just gonna do another one I guess change some of the values here um let me do 185 and I do producing that one as well so I've just produced two messages you say order ID 18 and order id19 here right so you can see some magic happening behind the scenes of um the rest proxy sort of checking things as we go out and check ask the address proxy if there's any new information and you can see one of them came in but um it will sort of go around and check information every minute go and sort of check for more information so you might just have missed one of them of course right so um that is basically how we bring the information into to service that so how much time have you got for enough because I know we had some technical issues so I'm just going to see how much all the other demos we can do I think we have another 30 minutes okay cool cool okay perfect okay there we go that was the other one as well so now you can see they both came in um I can I can sort of look at the executions here how they run and as you can with it all the flow designers and sort of track and see what was going on in in the flow Etc um so yeah but I'm not going to troubleshoot that I'm going to jump into stream connect and show what that looks like so hopefully that was quite clear about how to use the rest proxy of course um so let me switch screen and okay so um with stream connect um when you install that plugin you get a little menu so you can sort of see your topics you can rescan those topics you get namespaces and you get some other items here as well that we're going to talk a little bit about um so the first thing we're going to do is produce some some data so again I'm going to use flow designer to start with because I think it's the easiest way to look at this data and then we're going to look at other ways of doing it sort of scripting as well of course so um my script is called demo producer so again we're in flow designer we are going to read some records from this table that we talking about the demo sources table I'm going to read thousand records and we're going to create a payload and that's going to show you a little bit what does that mean creating the payload so in this case I'm just using the the payload Builder so take the record sort of yes drag my fields in here right as in however whatever payload I need to create and then I'm using that to to publish my message which you can see here as well but I said the message is basically the output from that demo payloader there Okay so to show that something is going on on the other side here I'm just gonna leave my rest proxy environment and yes jump down a few layers down here so I have a connector here what is called this container from there we go so I just have a consumer the system listen so I'm now going to approach from service now into um Kafka and on the other side which is my terminal window here I'm basically pulling that those messages down from Kafka as well right so um I might have to do that a little bit smaller so you can see okay let's uh kick this off so we've now taken 1000 messages from a table and service now and we pushing them to this uh this topic and that you can see as they're coming through as well so and that was um that was pretty quick um so that's using flow designer if we didn't jump over to do the same thing from script uh let's go let's go how to schedule a job it's quite a bit of space there um so again I just created a schedule job it just runs on demand and what it's doing here um it's basically I go and find the name of the topic or this idea of the topic by name I guess I go and get that fetch that um and then I read my thousand records as this from the same table we were reading from before and you can see how I build my payload and I can set my headers and keys whatever I want to do and then there is this API just to send the messages as well so hopefully that is straightforward I can just click execute now and same thing again it will just push all those messages you have to read it really quick as well cool um what else let's look at some um going they're sending data the other way so let me see let's see what we can do open another flow of course um yeah okay so with stream connect you get a new flow trigger as well so the new trigger is called Kafka message you select the topic it's a drop down so you can select the topics you have available and then you have some Advanced configuration here as well so you can choose to start processing from the end of the queue so we then connect to the topic you want to start reading from the end or would you want to start reading from the beginning of what data might be there right so depending on the use case here right we can choose that and then you can also sort of configure um the number of messages to process each run you go out and check as in we have a configuration here called automatically optimize so that will work out depending on your data and the flow as it runs it will optimize the way a number of messages we will pick up then right but there might be use cases where you need to just pick say 10 messages because you're going to send them on to some other subsystem maybe to say well I can only send 10 at the time for whatever reason um then you can override this and set your own number of messages as well uh cool so that's the flow today and then in the flow itself you can see we just use the for Loop so we now have an array of messages that we picked up um and then of course we do a log of that payload we parse it again and and then we have to write it to the demo orders table so I'm in a different instance so those records I generated before using the rest proxy they're not here there was a different instance you have to want to make that clear when you wonder why it's empty in there okay so I can um activate this one and then I'm going to leave that and in here I should have a few sample uh demo Scripts these are nothing fancy I'm using some Java code here um two years to write I'm going to see if I could find the the payload here but you can see order time order ID item id that's all I'm sending in a payload um so you will see what it looks like but let's do this sample demo and it's not the fastest here we go you can see how it's sort of started writing a few of them I think we start on one and we should do 10 of them or something like that right so it's not that much but for demo purposes that should work so this one is activated we can have a look at the execution for example if we want to have a look at that okay live demos you get some errors of course let's see if it's going to see if it clears out the killer beam that some um someone else puts some data into my my topic of course let's see what it looks like some other data okay it's saying like transform error kind of a transform error that means like it's not mapping to a correct field that's yeah that's exactly so the the thing because the way I've set up the demos I have here is that are using different payloads on the same topic so that that sometimes generates some funny issues right but you can see when the when the when the payload is correct which it is in this case for example it it works fine right so the other payload is you can't transform that right but it's just because the way I'm using my topics and sharing various types of messages across the like that but hopefully now I should have some data in here at least so you can see I got some data in here um let's see that's maybe it's still I might just move forward but um it could be that it was like a loads of data in there from from earlier okay it seem to have calmed down a little bit so that's good see if we can just run that again see if I can make it pretty let's review these guys okay officers okay so now I can see some of my messages coming through and then it looks much nicer right so you never know what you're going to get when you connect to a topic maybe maybe that's the what we should learn from this okay I'll move forward swiftly um so the next thing I wanted to show then was using the script consumer I'm going to leave it flow designer behind for a little bit I think that's deactivate this guy okay um right so if I go into consumers script consumer so similar things you you have a script consumer you can um it will listen to a specific topic I will show you how that works here in a bit but again I'll get my messages so it follows the same as we looked at with the flow designer right to get your messages I create my for Loop to process those messages um and then in this case I I write it to demo orders which is the table I'm using then right so so if you look at the bottom here we have uh what we call the Kafka streams you connect it to a Kafka stream and you can see it it tells me which topic it is um listening on and then it's not active so I can activate this one here there we go I have a similar script it's just it starts at 21 instead of one so that's basically the difference foreign and if we look at this now we can we can sort of see the subscriptions as well so we can see the data um being processed in a little while here but let's wait for it to be picked up give it a minute foreign so you can sort of obviously we haven't been running this this specific subscription as and I just activated it so you can see my my statistics here are not very useful at the moment but the longer they run the more accurate they get of course then right so you can see how you get input rate as in how much data is coming in over a five one and 24 hour period uh same thing with the sort of processing rate and the queue depth as well right and you can see at the bottom here as you consume messages and the offset and other good statistics that you might want to track as well and then if I look at my orders again you can see how I now fill in with some more data in here as well right so so that's the script consumer I asked quickly I'm not going to do the demos for the other consumers but they are going to show you what they look like because they work exactly the same way and so we have a transform consumer so you configure it the same way um we sort of which which topic to listen to and then I have my transfer map and the transfer map could just be as I said the existing transfer Maps you have to go and show you what that might look like in this case so here we are back in the Google transfer maps and demo order you can see I have my order ID and I can run my my transport Maps or my scripts Etc and whatever I need to do so that could be the same and just to jump to the ETL consumer as well works the same way you configure which topic you want to listen to and then um you have your Transformer here as well that that would run so um hopefully let's give you an idea of how you can use Kafka connect crafter servers now either using stream connect or using the spoke of course and um hope that gives you an idea of what you can do with this and how to implement it in in your environment foreign I think like and that's great and like uh oh I really loved like how simple it is using the floor designer itself right like you don't have to write a lot of code or anything you just have the flow that's there which is just sending the data as well as you're receiving it and also like for kind of like I think like the script consumers and other consumers will be used if you have an already existing data source that will be there right and then you can utilize those things so yeah that's that's great I think like Anders thank you very much for showing us what exactly is stream connect and all and folks who are watching us uh sorry about the technical issue this happens it's a virtual world a lot of times things happen so I think that's all we have so make sure guys you check out our Utah calendar so there are a lot more other Utah content that's there on Devlin dot SN slash Utah and also you know knowledge is coming up so if you haven't registered go ahead and register at knowledge.servicenow.com and make sure you subscribe to our Channel like the video and share with people who want to learn about steam cleaning that's all I have thank you very much folks uh we'll we'll meet you in another episode thanks Anders do you want to plug in anything enters before we no yes you already did knowledge that's perfect yeah see you guys bye thank you bye [Music] thank you foreign [Music]