×

We use cookies to help make LingQ better. By visiting the site, you agree to our cookie policy.


image

Programming, Analyzing Server Logs with Kibana

Analyzing Server Logs with Kibana

so now that we've actually got file beat

up and running in imported some access

logs into elasticsearch let's use Cabana

to visualize that data using the

dashboards that come with file beat

makes life a lot easier and it's pretty

fun too so let me show you how to get

started so while your server is still

running go ahead and pull up Cabana to

do that just go to 1:27 dot 0 dot 0 dot

1 colon 5601 and you should get to a

screen that looks maybe something like

this click on the management tab and the

first thing we need to do is create an

index pattern for our new file beat

index that we import it so click on

index patterns and right now you should

just have that Shakespeare pattern that

we used earlier in the course but let's

add one for the actual access logs that

we just imported so say create index

pattern and type in file beat dash star

and hit enter and sure enough it found a

file beat index that we can play with

and here are some of the fields that we

have to work with here it's good to

familiarize yourself with them for so

for example if you wanted to look at

response codes that would be under

Apache to not access dot response

underscore code so this is the structure

that file beat has imparted upon our log

data while importing it directly into

elasticsearch alright so we have that we

can make that our default if you'd like

and let's mess around with it let's go

to the discover tab and see what's in

there kind of explore some of the data

and you can see we're still in the

Shakespeare index we have to switch that

just click on Shakespeare and change

that to file beat now it's going to say

no results found and this can be very

confusing so don't let this discourage

you there really is data there the trick

is that you need to get the right time

range okay so up here in the corner you

see where it says last 15 minutes that

means that you're only looking at data

that came in during the last 15 minutes

and since we're using an older log file

that was generated in May of 2017

in order to see anything in cabaña first

we need to change the time field the

time range to actually encompass the

data that we have to work with so click

on that and now we can change that to

absolute and change this to the first

week of May so we'll go from May 1st

2017 to May 6 27 teen and hit go and now

after that does a little bit of

searching it should come back with some

actual results all right that's more

like it so this is more interesting so

we're seeing already here just a count

of how many documents came in how many

individual access hits or log lines

occurred over time during the state

range so you can see there was a big

peak of activity on May 5th around

midnight mmm I wonder if that was a

spider or a bot of some sort I bet it

was and you can kind of preview some of

the data here as well so if you want to

explore any individual time entry here

you can open that up and see what's in

there ok so from here you can do

searches so for example if you did want

to search for a 500 you know server

errors for example you could copy and

paste this Apache to access dot response

code here for example and the syntax up

here is just field name colon whatever

you want so that will give you back all

the 500 errors and you can see that they

occurred in the Big O little spike on

May 5th

well let's pour that more deeply in a

moment you can also just click on these

things too and say add and automatically

take a look at it that way too

and explore it in more depth so if you

want to look at individual 500s that

would be a good way of doing it you can

see that this one's coming from Beijing

China and this is why we installed that

geoip plugin for elastic search by the

way it allows you to see very easily

where these IP addresses are coming from

which can be useful for tracing down

where issues like this or originating

from this one came from Mountain View

North America so probably just Googlebot

or something take a look at another one

here Kiev interesting all right but you

know looking at textual results is all

well and good but I want pretty charts

and graphs you know that's what we came

here for I want something that can give

Google Analytics a run for its money

only it's for free and I run it all

myself the great news is we imported a

bunch of dashboards as part of

installing file beat previously so let's

use them just click on the dashboard tab

here and look at all these goodies that

we have if we go back to the dashboard

main dashboard page here you can see

there are many to choose from we are

using an Apache to log so we're gonna

click on file beat Apache - - dashboard

and check that out

so this looks you know pretty useful

we're seeing a nice little chart here of

all the hits per time per minute here

from the blog data that we're using in

the date range that we specified and you

can drill in on these you can see

there's stacked bar charts so they're

actually telling you broken down by

access code what's going on here and you

can see there there's that little

anomaly of 500s there that's sticking

out on top of that particular time slot

on the midnight hour of May fifth and

out here you can see things like how the

request codes break down you can see we

had a little bit of a flurry of 301's

at one point this is how the access urls

break down you can look at the actual

breakdown of browsers that are hitting

your data that's kind of cool too broken

down you know by Chrome and individual

versions of Chrome on this outer ring

for example that's how to read this one

so this inner ring is the browser type

and then we have individual versions of

that browser broken down outside of that

ring as well you can also take a look at

what operating systems are hitting your

website with this little graph here you

can see that other is pretty popular and

that's because BOTS select my site as

much as humans do but you know all sorts

of cool stuff and if we had configured

things to import error Lots as well we'd

see even more stuff you know since we're

only dealing with an access log in this

particular example we're only seeing

half of what this has to offer but you

can see it's pretty cool stuff you can

also see that there's a hotspot here

looks like it's in Moscow so for example

you can also drill down here I mean the

best way to learn this is to just play

around with it so I encourage you to

just play around with it let's drill in

on that 500 spike there that we had you

can see if I click on this individual

piece of the stacked bar chart up here

it prompts me to apply the following

filters so do I really want to narrow

down my visualization to 500 access

codes during this particular hour yes

yes I do and now we can see that this

flurry of five hundreds came during a

very specific time range between 7:15

and 8:15 of may 5th 2017 and furthermore

it came from these places so you know if

we actually click on that we can

actually expand any of these by the way

to get a closer look so let's go ahead

and open up that map and we can drill in

and see that that's actually coming from

Moscow I'm not making this up folks

someone from Moscow was hitting my site

hard this at this time and actually

crashed my server for a while so

conspiracy probably not it's probably

just some spider some innocuous search

engine because because frankly this

website isn't scaled very robustly so

probably not evil it's probably my fault

for not having enough capacity but

anyway there you have it I'll begin

closed back out of this shrink it back

down and remove these filters by just

hitting the trashcan here and we're back

to where we started and you can put the

date range back to whatever you want as

well so fiddle around and in our next

lecture I'll give you a little bit of a

challenge to try and find a specific

answer to a specific problem so let's

try that next

Learn languages from TV shows, movies, news, articles and more! Try LingQ for FREE

Analyzing Server Logs with Kibana Análisis de registros de servidor con Kibana Kibanaによるサーバーログの分析 Analisar registos de servidor com o Kibana Анализ журналов сервера с помощью Kibana Kibana ile Sunucu Günlüklerini Analiz Etme Аналіз логів сервера за допомогою Kibana 使用 Kibana 分析服务器日志

so now that we've actually got file beat

up and running in imported some access

logs into elasticsearch let's use Cabana

to visualize that data using the

dashboards that come with file beat

makes life a lot easier and it's pretty

fun too so let me show you how to get

started so while your server is still

running go ahead and pull up Cabana to

do that just go to 1:27 dot 0 dot 0 dot

1 colon 5601 and you should get to a

screen that looks maybe something like

this click on the management tab and the

first thing we need to do is create an

index pattern for our new file beat

index that we import it so click on

index patterns and right now you should

just have that Shakespeare pattern that

we used earlier in the course but let's

add one for the actual access logs that

we just imported so say create index

pattern and type in file beat dash star

and hit enter and sure enough it found a

file beat index that we can play with

and here are some of the fields that we

have to work with here it's good to

familiarize yourself with them for so

for example if you wanted to look at

response codes that would be under

Apache to not access dot response

underscore code so this is the structure

that file beat has imparted upon our log

data while importing it directly into

elasticsearch alright so we have that we

can make that our default if you'd like

and let's mess around with it let's go

to the discover tab and see what's in

there kind of explore some of the data

and you can see we're still in the

Shakespeare index we have to switch that

just click on Shakespeare and change

that to file beat now it's going to say

no results found and this can be very

confusing so don't let this discourage

you there really is data there the trick

is that you need to get the right time

range okay so up here in the corner you

see where it says last 15 minutes that

means that you're only looking at data

that came in during the last 15 minutes

and since we're using an older log file

that was generated in May of 2017

in order to see anything in cabaña first

we need to change the time field the

time range to actually encompass the

data that we have to work with so click

on that and now we can change that to

absolute and change this to the first

week of May so we'll go from May 1st

2017 to May 6 27 teen and hit go and now

after that does a little bit of

searching it should come back with some

actual results all right that's more

like it so this is more interesting so

we're seeing already here just a count

of how many documents came in how many

individual access hits or log lines

occurred over time during the state

range so you can see there was a big

peak of activity on May 5th around

midnight mmm I wonder if that was a

spider or a bot of some sort I bet it

was and you can kind of preview some of

the data here as well so if you want to

explore any individual time entry here

you can open that up and see what's in

there ok so from here you can do

searches so for example if you did want

to search for a 500 you know server

errors for example you could copy and

paste this Apache to access dot response

code here for example and the syntax up

here is just field name colon whatever

you want so that will give you back all

the 500 errors and you can see that they

occurred in the Big O little spike on

May 5th

well let's pour that more deeply in a

moment you can also just click on these

things too and say add and automatically

take a look at it that way too

and explore it in more depth so if you

want to look at individual 500s that

would be a good way of doing it you can

see that this one's coming from Beijing

China and this is why we installed that

geoip plugin for elastic search by the

way it allows you to see very easily

where these IP addresses are coming from

which can be useful for tracing down

where issues like this or originating

from this one came from Mountain View

North America so probably just Googlebot

or something take a look at another one

here Kiev interesting all right but you

know looking at textual results is all

well and good but I want pretty charts

and graphs you know that's what we came

here for I want something that can give

Google Analytics a run for its money

only it's for free and I run it all

myself the great news is we imported a

bunch of dashboards as part of

installing file beat previously so let's

use them just click on the dashboard tab

here and look at all these goodies that

we have if we go back to the dashboard

main dashboard page here you can see

there are many to choose from we are

using an Apache to log so we're gonna

click on file beat Apache - - dashboard

and check that out

so this looks you know pretty useful

we're seeing a nice little chart here of

all the hits per time per minute here

from the blog data that we're using in

the date range that we specified and you

can drill in on these you can see

there's stacked bar charts so they're

actually telling you broken down by

access code what's going on here and you

can see there there's that little

anomaly of 500s there that's sticking

out on top of that particular time slot

on the midnight hour of May fifth and

out here you can see things like how the

request codes break down you can see we

had a little bit of a flurry of 301's

at one point this is how the access urls

break down you can look at the actual

breakdown of browsers that are hitting

your data that's kind of cool too broken

down you know by Chrome and individual

versions of Chrome on this outer ring

for example that's how to read this one

so this inner ring is the browser type

and then we have individual versions of

that browser broken down outside of that

ring as well you can also take a look at

what operating systems are hitting your

website with this little graph here you

can see that other is pretty popular and

that's because BOTS select my site as

much as humans do but you know all sorts

of cool stuff and if we had configured

things to import error Lots as well we'd

see even more stuff you know since we're

only dealing with an access log in this

particular example we're only seeing

half of what this has to offer but you

can see it's pretty cool stuff you can

also see that there's a hotspot here

looks like it's in Moscow so for example

you can also drill down here I mean the

best way to learn this is to just play

around with it so I encourage you to

just play around with it let's drill in

on that 500 spike there that we had you

can see if I click on this individual

piece of the stacked bar chart up here

it prompts me to apply the following

filters so do I really want to narrow

down my visualization to 500 access

codes during this particular hour yes

yes I do and now we can see that this

flurry of five hundreds came during a

very specific time range between 7:15

and 8:15 of may 5th 2017 and furthermore

it came from these places so you know if

we actually click on that we can

actually expand any of these by the way

to get a closer look so let's go ahead

and open up that map and we can drill in

and see that that's actually coming from

Moscow I'm not making this up folks

someone from Moscow was hitting my site

hard this at this time and actually

crashed my server for a while so

conspiracy probably not it's probably

just some spider some innocuous search

engine because because frankly this

website isn't scaled very robustly so

probably not evil it's probably my fault

for not having enough capacity but

anyway there you have it I'll begin

closed back out of this shrink it back

down and remove these filters by just

hitting the trashcan here and we're back

to where we started and you can put the

date range back to whatever you want as

well so fiddle around and in our next

lecture I'll give you a little bit of a

challenge to try and find a specific

answer to a specific problem so let's

try that next