we are rolling all right well everybody Welcome to beard with Engineers session 21 uh last one of 2023 um which is kind of crazy to think about um we're really excited to be with you guys all here today um it's you know it's always like getting together to hang out with with my friends so appreciate you guys all joining us um so we'll kick off again um as always with the uh the Safe Harbor statement right we talk about a lot of stuff in here some of it may or may not be released yet some of it may be on a road map some of it might not ever come to fruition so don't make any stock purchasing decisions based on anything we say here leave it here all that fun stuff um so we always have a very very informal agenda here um we'll just kind of walk through this bit by bit today um and we'll we'll kick it off with why are we here and what's the goal of beers with Engineers so um why did we start beers with Engineers um it was really you know will and I got together and thought not only does not a lot of people know that service now has any sort of cloud native capabilities um but what we found was that as we started talking more and more customers about it they all kind of felt like they were alone on this like road towards how do we integrate all of this stuff and so felt like it's a good idea to build a community together of similar customers dealing with similar problems and providing valuable content around what are the capabilities and give people an opportunity to have discussions around what they're doing what's working what's not working what the struggles are so that's why that's why we're here so who we are uh I am uh the manager of the Enterprise applications team here at drw um I once upon a time was a service now employee um I have joined the dark side as a customer um and uh honestly really really enjoying it um I am an old school nerd from way way way back in the day um and uh I always joke that if it's got a one or a zero in it I've probably Managed IT at some point in my career I was actually just thinking about this the other day the first time I learned hexadecimal was actually to Haack a Dungeons and Dragons video game so that I could have Max stats that's that's old school um so I am not going to be drinking a beer today because I'm still not feeling 100% but I am drinking water um to try and help counteract that so um I love Tech I love to solve problems with tech and I love spending time with my family playing a bunch of board games and of course training Jiu-Jitsu as much as I can get away with will thanks sir everybody will hallum H advisory solution architect focusing on itom at service now been in technology for a good while and and um really focusing lately on automation especially in the cloud native Arena get a lot of satisfaction out of automating repetitive tasks in my spare time I I like to hang out with my family I play some pickup hockey and and video games to pass the time today I will be drinking a juice jolt extra juicy IPA from Southern Tier Brewing Company uh and also with us today we have a guest speaker a teammate of mine Taran would you like to give a brief intro sure thanks for having me today um my name is tarish and I'm a solution consultant with service now Cloud observability team um I joined little over a year ago um I'm actually taking the opposite of what Mike mentioned that I was at the customer before and then I joined service now um a year ago and it's been uh great to to be here awesome well thank you so much again for coming to talk to us today about cloud obser cloud observability logs fast yeah exactly so just to kind of lead off into what Taran is GNA go into we've had Cloud observability formerly called light step for a while we did a session on that a few months ago and so now what this is bringing to the table is the ability to take the same easy to set up ingestion methods that we had in place for metrics and traces and had the capability to take log data and add that to the mix as well so I will uh release my share and uh hand it over to you awesome thank you let me know if you can see my desktop now aw looks good good uh okay so as I said my name is Taran I'm happy to be here we will be talking about logging with service now Cloud observability today um who loves uh going through slides but I do have some slides to go through first I just want to make sure that um everybody here is on the same page before we jump into a demo that I will have at the end of the the session I don't have 25 slides or so it's just um eight slides I think to go through and and then we will jump into a demo uh so let's just talk about like you know what is the big deal with logs logs are everywhere I'm sure if you're in technology you've already heard about logs or working with them on daily basis they're emitted by applications and infrastructure and help to have context regarding application um application performance um you might have heard about the the three pillars of observability which are logs metric traces uh that help us understand the root cause and underlying dynamics of the issues so logs are one of the important part of that uh three pillars and in addition they are the most basic form of telemetry data um which you don't need to have any you know code changes to to get the value out of that uh let's talk about or start actually from how logs are being collected um you know if you're in technology you might have already known this but I have just a slide to talk about how they're they're being collected um every device and application in the network it generates log data those logs get ridden into a log file uh into a disk and then the logging agent which will be installed on on the machine will read the log file in disk and sends it to uh your choice of you know logging provider um the the agent that is installed in the machine uh not only collects and filters the log but it also parses uh the logs and converts them into the formats before uh they're being forwarded to the the logging provider which I will talk about the the agent in in the in the next slide or two um similarly the the architecture for cloud native style is is almost the same except that the logging agent is probably outside of your cloud provider an example of an open-source logging agent is open to limitary collector or you know some customers are using Vector but we also have some vendor specific uh logging agent um such as data dog agent which is proprietary uh to that vendor specifically but if you're using open source um such as you know open Telemetry collectors you would be able to send send your logs to any logging provider of your choice so you do that one time and then you would be able to ship logs to any provider that you want so once you collect the logs you can use the logging agent to convert them to the format that is acceptable by uh by the the logging provider in the case of Hotel uh you will be using Hotel collector um to receive logs make some modifications and then export them as OTP format um and in in our case to send it to service now cloud observability logging uh right now we accept um OTL format elastic search and also data dog agent format so if you already have these in place um it's just a matter of you know five 10 minutes to ship logs to us which is which makes it easy to start with if you're uh in the beginning of your observability journey and you want to start from somewhere usually find that it's easy to you know start a PC with logs and then just go from there so just looking inside um uh the agent here um as we talked about the the logs are being WR written into the the log file and then agent is basically just a you can think of it as a pipeline that it receives the data it processes the data and then it exports it to loging provider so basically helping you out on getting the logs from the system do some processes uh usually it's resource enrichment or changing the you know the parsing to change it to the fields that would be useful for your business do sampling if you have a lot of data usually we see customers do you know 20% 30% sampling rather than keeping everything in the system and do some modifications to it before exporting it to to log providers usually it makes it easy for your log provider once to accept it and also being able to uh have the fields in a structured way rather than being unstructured and if I remember correctly you can use that capability to do for example you could take a certain rex that might contain like a social security number or some other personal identification information and um mask it or replace it before sending the log the rest of the log over to Cloud observability right exactly you can do so many things with you know sampl to your point like modifications if you want to mask data usually if you know there's a if there's some compliance I've worked with uh customers during the PC's that they're due to compliance um regulatories in place they were not able to send us you know customer IDs like detailed information about customers that is something that will happen in the agent layer that you don't want those those logs to to be shipped to the the provider but you can also do enrichment to the resources modifications um to help on basically what your business is looking for um to make sure that the logs are being valuable during your incident resolution um and investigation cool so moving along uh next slide um so now that we have a baseline of hogs are being generated and and and collect that let's talk about our our approach and service Nows Cloud observability um log offering um Safe Harbor I think Mike went through that but it doesn't hurt to have a second one U just to make sure that things that we are talking about might be forward-looking um so I have this slide I wanted to talk about what we generally hear from customers regarding challenges with with logs are one that businesses and especially the the big ones they generate large amount of data daily that ranges from you know 10 15 terabytes of logs per day to we already well we also had cases for 200 terabytes a day and more um so sorting logs becomes very expensive storing logs becomes very expensive uh with new applications being gathered every day the complexity of uh Cloud native applications um as well as compliance and regulatory requirements to keep the logs for a longer period of time um and the other on the other hand also to make data manageable we see customers have to make a decision between between either limiting the data uh that are being stored or reducing the retention period so that it's um as short as you know few days or few weeks um Howard this can make this difficult to identify security issues in in modern architectures like microservices and containers um you know with multiple Cloud providers and such um the Second Challenge is also the scalability without compromising performance and search experience for for end users we hear this a lot from from um customers that you know they have large amount of data they want to um search around it and it takes them a lot of time to even just for the query um to to respond so we build a solution that is one cost effective and then also performant to address both of these uh these challenges and the third one is on top of everything we also added the third element as a cherry on top um to have have everything included as part of one ecosystem in service now um so you have the best world of observability solution with logs metrics traces in addition to your itsm for Incident Management and event management with a Ops capabilities for for advanced alerting so everything is part of this ecosystem of service now um so that you can have better correlation better integration between uh different systems that uh will will make the investig ation and you know it will reduce your mttr um and customers often ask us how we keep our cost low and and the answer is logging storage is built on a modern architecture that we have at service now which is purpose built to be efficient and and cost effective um and we built this from from scratch um so so if you look at the the lift diagram for legacy systems often time the computer and storage are combined that will impact the performance so you're basically you have a VM and storage that are in the same machine they do the data ingestion data storage processing query handling everything is in one um however in in our modern architecture we separate ingestion from the storage and from queries and different tiers uh so basically this will give us the opportunity to have those uh independently scalable um as long as you know you have lots of data you can store those in in um in um object storage for like you know Cold Storage retention so we are not compromising performance for that and I had personally I had PC's that people will um vow people with with our performance of the UI with lots of data that they're sending us um and a little bit of history about our logging solution which is it used to be era Software founded by Todd person um who joined us through era acquisition back in 2022 with his team um which is right around uh the time that I joined also service now and it's been an absolutely amazing to see how fast our engineering team um have been able able to integrate era into service now cloud observability in such a short period of time um and make it available for for commercial use um so we have have a great team of Engineers and leaders who know the market really well and they listen to our customers uh to tailor our products based on their needs let me know if there are any questions that we can an we can um answer I know that I'm I might be going a little little bit fast here yeah as always everybody's invited don't be shy feel free to shoot questions into the Q&A or just come off mute and and Shout them out as you as you prefer agree awesome cool um so I wanted to talk about a little bit about hot storage Cold Storage I'm not sure if everybody is familiar with these terms but um uh hot storage is basically data is immediately available it's queriable you know think of it as like living in a PO you have access to to the data Cold Storage it it allows customers to keep the data for longer period of time and we use object storage to store very compressed data which is significantly cheaper um but the downside is it's not available for search and query right away so if you're keeping Aller data in cold storage and you know let's say there's a security breach and you want to look into the data that is not in your hot storage um time frame hence it's move to Cold Storage uh we have a feature that's called rehydration uh you might have heard other vendors also have the the same capability but uh our rehydration capability is available through UI so you basically just select the time frame that you want to bring back the data um and you can bring the data back from uh Cold Storage to hot storage which for us is just moving the data so no indexing involved no extra charges for that um you can just move your data to hot storage do your investigation potentially create a notebook or or you know dashboards for anything that you want to look into that data and then release the data back to to Cold Storage when you're done um there are also events such as you know we work with customers that uh during the events like Black Friday or Super Bowl which they had before and they have the data from like a year ago um and there's a during those events usually there's a big surge of uh usage that customers want to analyze the logs um or metrics from a year ago and predict what resources they would need to be prepared for this time so this is a great example of you know you would be able to based on your retention policy you would be able to bring back those logs if you've been um saving those logs for uh for you know let's say a year or so in addition there's another case also for uh PCI compliancy sometimes you know Enterprises would like to keep the data for a longer uh period of time you know 90 days or even a year and those cases are where you would be able to um store your data in cold storage for and and don't pay a lot of money for that uh versus a hot storage so being able to benefit from cost reduction and the flexibility to access old logs uh would be really beneficial with cold storage and and rehydration and uh one thing to point out that I have it in this slide is for us rehydration is fast and efficient uh what I have here says you know the search for one or to bring back one terabytes of data from Cold Storage it takes 12 minutes uh but I have to point out that this one terabytes of data is highly compressed data so you can think about it as if you know three terabytes of uncompressed data if you want to bring it back it's just going to take couple of minutes for you and the balance both the balance of hot and cold as far as how much hot storage versus cold storage is provisioned or entitled plus the uh retention that's um that can be adjusted based on customer need and I believe it's available at both within the existing Cloud observability kind of um taxonomy it's available both at the organizational level which is generally like your top level organizational container or at the individual project level and the project is kind of the basic um kind of the basic swim Lane for a set of observability data and so aligning to both of those you can get it customized as far as how long it's retained and how much is hot and how much is cold yeah exactly to jump in there's a couple of questions in the Q&A we want to hit those really fast yeah let's do that um so let's see Durga is asking if we have an existing logging solution in place how easy or complex is it to share the logs with service now observability um yeah I I would say it's it should be relatively straightforward based on the the fact that we emulate the elastic and data dog um U apis as well well as the native open Telemetry what what would you what would you think yeah that that's right exactly well as you mentioned if you already have a logging solution in place there's a good chance that you are using a logging agent right you know if in the case of uh data dog if you're using a data dog agent or uh we see customers using you know Vector fluent def fluent bit you would be able to just use those logging existing logging agents and send it to us our recommendation is using open Telemetry because uh you will adhere to the semantics of open Telemetry from like you know log attributes and Fields and metadata that it has um so it just makes it easier to accept those logs as structured but even if you don't have open Telemetry collector and for some reason you're not able to stand that up or if you even you know don't want to do a lot for sending the logs um if you already have a logging solution in place I would say it's very easy we have the configuration also the documentation is in in our uh website regarding the configuration that we provide for each of these types of um uh agents to to be able to just send the data to us I did include a link to our documentation page that talks about the different log Integrations and how to configure to send logs from existing logging Solutions into Cloud observability so that link will be available in the slides that go out with our uh with our wrapup email as well uh the second in addition to that question um is that same logic applicable for metrics and traces as well if we have an existing APM agent solution is that metrics and traces can know flow through the absorbability without adopting the oel yeah for metric ENT traces is a little bit different if you if you're talking about um you know let's focus on metrics first um if you're using AWS and Azure we already have integration for almost everything like you can integrate into Cloud watch and then you would be able to send us any all of the metrics that you have same thing for for Azure if you're if you're using Azure monitor we have integration so a lot of Integrations and we have the the list of all of the Integrations that we have for metrics in our documentation um as long as you have uh we support these Integrations you would be able to send the metrics in the same way um and for for traces um the we don't have for service now Cloud observability the main way that we accept traces are uh through open Telemetry uh so we don't have any proprietary agents um it depends on you know we have some customers using Jagger some customers using you know Prometheus and such so as long as if you're using open Telemetry uh you would be able to send us your uh your traces uh for other I'm not sure if you have anything in mind and um I don't think we want to go deeper here there uh but traces I would say it's the if you are not using otel then that would be a conversation for us to see if we can you know help you out to uh stand up a collector to change the um to use the open tetric collector to change your format to OTP because that's the way that we will accept it uh but this is what we've seen the organization and Enterprises are you know mostly they're going um into open Telemetry and this is the method that we have yeah and that kind of goes along with the theme that we've been hearing which is when it comes to those three pillars of observability logs are kind of the easiest to gather right because they're they're already there you just got to send them somewhere and then comes metric and then comes traces so there's definitely that kind of climbing level of difficulty when it comes to those three pieces of observability data um I can just add from my some stuff that I've been doing just uh in my lab that um a tools a tool like cribble for example can in theory take a something that's not open Telemetry and turn it into something that can be delivered to an open Telemetry endpoint so even if you've got kind of more proprietary data it's not out of the question it's just a question of how much effort is involved to put it into a form that can be ingested by open Telemetry if you've already got o otel instrumentation out there you know that that's obviously the the easiest to just you can just Fork off a copy of that send it to service now cloud observability and you're Off to the Races and then kind of as you back away from that it just kind of ratchets up what it would take to make it happen is there any road map at least to dink that from hotel agents for traces because right now we have an existing APM agent solution but it is not coexisting with the otel um agents and there is a challenges in in making coexist and have the Tres separated so right now the other thing that we have is we also accept um traces from data do agent but except from that um I'm not aware of anything that we have in the road map for now um and and as I said if you know if there are cases for customers that they're looking to see if they can use their current agents to use us the traces we can definitely involve our architecture to help you out um but the main uh Remains the the open toiletry for now I appreciate you thank you okay one more question um where are logs stored is this not built on public Cloud you're running custom DBS uh it is a it's in public Cloud um so the logs are stored in in gcp um and you know obviously it's a um it's a it's a SAS solution that we offer so there's no installation and on Prem and such um I hope that it basically lives everywhere that cloud observ it's it's part of cloud observability right so wherever the um formerly light step now service now cloud observability is hosted that's where the log facility is also hosted right yes that's right and will I think you were mentioning something regarding um you know hot and cold retention from from pricing point of view um just had slide to highlight that our pricing is only based on ingest and hot and cold retention so to your point the the hot and cold retention are separate and you can configure that so when we dealing with contract with customers we usually ask about the requirement regarding how much they want to keep the data in hot versus cold retention um you know obviously usually we see three seven 10 days for hot retention because that's where um how much time you would need to do your investigation but uh for any you know PCI requirements or compliance requirements and such or you know the cases that I mentioned regarding Black Friday big events um you might need to require to use cold storage to keep the data for uh for a longer time uh but we don't have any limits or charges on you know how many queries you're running how much it's taking for queries to run um the cold storage retrieval you know how much data you are bringing back to hot storage and also number of alerts and the reason I'm bringing these up are uh because they're that we have some vendors that are charging for these I just want to make sure that you know people understand that the pricing is solely based on inest and hot and cold retention awesome and I think that was it for slides before we jump into the demo I just want to make sure if there are any questions we can answer if not we is clear yep awesome cool cool okay so um I hope you can see this environment that I'm showing this is our uh this is our logging page we I'm inan Cloud observability um environment uh which you know we have service directory if you're sending services and such we have also notebooks um similar to you know not notebooks that you've seen probably in other providers regarding uh you would be able to do your investigation obviously we are not going to touch on those today I'm just going to touch on the log section um that we have for uh for the platform uh there are multiple areas that you can do search you can see on the left side these are all the fields that are coming in um in the logs um so basically you know you use the collector or your agent to identify which Fields you want to include in your logs if the logs are coming in as structur then you would be able to capture these such as you know if you're using open Teter collector these all will come through uh through open Telemetry uh semantics but also if you want to add any custom field like you know customer ID or anything that would be helpful for your business then that's where uh you would be able to just see the the list of all of the fields that are here you can search based on these fields let's say you know I'm looking for service name um and then here I can say I want to include or exclude this one or search on the field names also um and as you can see the um the filtering is so fast um obviously this is my demo environment I don't have much um you know a lot of data on it but the same experiences as if you're sending 10 15 terabytes of data every day and then we also have fre text search also on top so if you want to look into errors it looks looks into um you know in the body and these are the log lines uh log lines that we have here um and um if you click on one of these log lines it will bring you um the information regarding um the fields that's similar to what you see here but these are also clickable so you would be able to you know for some of these that are you're we are able to uh filter you can say show matching hide matching or add them to the the color color column as this in the table um and then you can you know just click on it and filter any of the logs based on this as well we also have um span ID and um you can see that span ID and Trace ID is also included is as part of the log so this is a uh if you want to have correlation between your logs and traces this is how you would do that you would include your span ID and Trace ID as part of your logging instrumentation to send it over so we will we will correlate those traces also for where where there's a logs um and you will see that this open link Trace will become um will become enabled if there's no tracing available for it this would be disabled and if I click on this one it will bring me the actual Trace um I know that this is this would be very helpful during investigation if you're seeing something in logs and you want to see exactly what's happening you can go to the span you see that the span has errors um and then you know this is showing that the the trace has 15 spans and if I go through these spans it provides me more information regarding um what's going on with this um you know piece of application or service uh for the issues that are happening so that you can figure things out faster and I have this one I think it will provide more information so for for this span it says you know the resources exhausted so you will have the the logs here as well to provide you more information I can also jump back to the logs so let's say I start with tracing for my investigation and I want to see the logs to provide me more context surrounding what the issue is I can also just say view logs and then I would be able to just jump into the exact logs for the period of of time um you you notice that it brings U just couple of log lines for that but you know let's say you want to see these logs in context so I will just say see in context and it will show all of the logs before and after this event also um so basically you know around that time frame um it's you're able to see before and after uh what has happened in the logs uh to be able to provide you more information um and then what else I can show you here the raw log here also you can see how the fields are coming in from the logs that are um becoming these fields um that you can query and um you can also search based on through those um obviously you can identify which columns you want to see based on the data that is coming in um and the information for those we also have um live tail which you want to see the logs as they're coming in uh sometimes during the investigation you want to have the log uh live logs that are coming in and this is the option that you would be able to see those information or if you want to search based on you know specific like 60 Minutes last 60 Minutes or 4 hours or a custom window to uh filter based on the logs this is how you would use those got a couple questions in the Q&A uh one from an anonymous attendee who says uh keeping the otel context in mind why would I relate logs to spans versus using span events logs to spans versus span events um I'm not sure what do you mean by span events um but the correlation between logs and spans usually sometimes you know you you get a lot of information from from traces and spans uh but sometimes uh there are you know authentication issues sometimes that are not uh if if if you're not able to see those in traces logs might provide more context um or additional context around like when this happened um usually you know since since it's based on timeline logs might provide more information regarding did I start seeing you know some warnings before this issue happened versus the spans and traces would help you identify the root cause of the issue um there's probably a bit of kind of um economy at work as well right because I'm I'm guessing in order to generate a span event you'd have to include that in your instrumentation that for this span I want to generate an event it's almost like the traditional kind of monitoring use case versus something like like log analytics where for traditional monitoring you have to know what you're looking for you have to know where things might go wrong whereas if you just adopt a standard log line format that includes the span and Trace ID you don't necessarily have to um be preent in predicting where things might go wrong and adding an event around where the error may occur right you can just have it stand ardized in your log format and then wherever the error occurs if there was a if it was within a a a span if your logging standard says if there's a span in play put the ID in the log line then um and it's kind of you know that like like all things in it there's multiple ways to go about it and it's more just what's appropriate and most use for your particular situation I would think yeah I think you're a spot on also and and as I'm looking for span events I I get what they're saying sometimes this this information shows up here and for some of these you don't see the the span events here um so I think it's just additional information through logs that you would be able to to um identify versus what I can see here um and as I said traces will provide you the root cause of the issue and how you know what's going on versus the the logs usually there's a you know timeline for when things started going south and what's going on with with the uh with the system or if there's any additional stuff that's happening but usually yeah the traces provide more information for you well and the nice thing is with this scenario is you have all three in one tool so you don't have to Pivot even if it is um perhaps there were a multitude of signals there were span events and there were log entries and in this environment you can see all of those juxtaposed with each other and you don't have to just constantly swivel or tabs tab switch between different tools to kind of correlate those separate sources it brings it all together in one place which um you know thinking back to my days as a customer I would have found pretty darn useful yeah and I think it's you're absolutely right regarding we do the correlation uh in the platform so you don't have to go different sections of the platform to figure that out the reason we have logs is you know we know that some people would like to just go directly to logs and start doing their uh search regarding you know errors and what's happening and such but for the rest of it you see that there's no in the platform we don't have different section for traces you know metrics separately uh what you will see a service directory that we list all of the services uh but we also have dashboards that you can have all Telemetry datas in in one place and we have um query language unified query language that you know we have builders that you can um I can actually show you an example of that that um you know you can use either logs metric traces uh to basically instrument have the dashboards and graphs wherever however you want to do that either you can use the the Builder or you can use the editor if you're more um expert on writing queries uh but it just makes it easy you can just say I'm looking for all Telemetry data or I'm looking for metrics traces or logs separately uh but yeah the nice thing about it is like you can have everything into One dashboard um I think this one specifically is Production service for logs um and you know if you're looking at service logs by Services um Errors By service logs you can just click on this one and then say search logs and it will take you to uh the UI that we saw for logs so basically from any dashboards you can just go to metric logs traces uh which to your point it does a correlation together to bring you the information and um ultimately reduce your mttr regarding what's going on in your system if I remember correctly we have our own Universal query language that we use for all of this is that is that right yes that's right it's called um unified query Builder and unified query language um and this is how it looks like so um you know you would be able to choose what Telemetry data you're looking for is it metric traces or logs or I don't know I'm just going to choose you know all Telemetry and then you would be able to say you know for for metric it will bring you all of the metric you can choose from um you can filter them aggregate them group them by group or add a formula that you want so there are multiple ways that you can uh slice and dice the the data however you want whether it's metric traces or logs so there's there's correlation between these and the back end um but we also have editor that you know if you um you're familiar with our query language then you would be able to write more sophisticated and complicated complex um uh queries we also allow you to Define these in terraform so there's a terraform provider for cloud observability so you can actually you know if you're if you're a very devops and cicd driven shop you can actually Define all of these dashboards and queries within cloud observability in code in terraform and then use the terraform provider to provision them into your environment which I thought was a pretty a pretty cool feature yep uh let's see another question in the Q&A um Matt Morris is asking this a very important question how do I get my cloud observability project opted in to use the log capability uh so I'm not sure if you're a service now customer but service now Cloud observability entitlement is separate from you know if you have itom or itm uh so basically uh the with licensing you you have to have licensing for service Cloud observability to be able to uh to use this one so it would be kind of talk to your pro talk to your accoun team to get yeah yes there are ways if you're using itom and you have Discovery license we have um open to limitary SGC uh that is part of your itom license so you would be able to send the information through Cloud observability to service now if you want to use it for you know improving your cmdb with Cloud native CIS uh use it for service now learning and event management that is included in your um itom Discovery uh entitlement but you won't be able to use the the service um Cloud observability Bui that I was tell that I'm showing okay so so the kind of um builtin Cloud observability entitlement you get with itom is just to use with it's basically just to use with the SG connector and it doesn't include logs so you need an actual Cloud observability entitlement in order to turn on the logging capability is that yes yes that's true but if they already have that is this included or is this a I'm sure is an additional cost based on the volume and size and the pricing that we were discussing earlier yes exactly so for for metrics it depends on how many metrics you have uh what is the retention policy that you want to have for for metrics uh for traces it's the same retention policy how much data volume would be for your traces um and then logs we talked about retention for hot and C so each of these they have different um different subscription units cool and just a little plug we did uh SGC oel session a while back so if anybody's interested you can go look that up on the YouTube channel we we got pretty deep on it that is true we're covering all the observability topics here at beers of Engineers um one thing I I can imagine my uh service now customers asking um so we've had a health log analytics offering for a while um at some point are we gonna provide a way to send things from this facility into HLA yes they're very exciting stuff happening regarding connecting HLA to Cloud observability logging um specifically to be able to use cloud observability as your log management tool and then ship those logs to HLA for health log analytics uh for you know basically the a Ops on your logging um I've seen demos it looks great regarding you know how you will set things up is just click of a button and then you would be able to ship the logs to HLA um but um I believe it's going to be available in February time frame the first release is going to come out in February uh so we we are very excited about that we have a lot of good stuff happening in the road map um uh which we've been working on so we'll have more updates maybe in January February time frame we can talk about that too excellent hug yeah and is that uh do you know is is that going to also still keep the mid server kind of in the middle the way HLA works today so I going send it directly to the back end yeah no mid server that's that's what we've been working on to be able to have the direct connection so basically what it will what you need to provide is your service now instance and the credential for your service now instance and then logs as long as you have it in uh service now Cloud Ops the connection would be direct connection no m server nothing else is needed oh wow that's that's huge yeah that that's going to be a big win yes looking forward to hearing more about that sure okay um let's see any other any other questions the Q&A is clear at the moment anything else on observability logs no all right awesome all right well we're gonna guess we'll we'll close out our recording and then we'll uh shift into our open Forum segment of the program uh Tarn thank you so much for coming and showing us the cloud observ ability one of these days I'll be able to get that word the first time uh Cloud observability logs greatly appreciate it s thanks for having me hopefully it will it's um it's been valuable uh for for audience and let us know if you have any questions perfect