Unsolved
This post is more than 5 years old
6 Operator
•
14.4K Posts
•
56.2K Points
0
1559
January 20th, 2014 10:00
Your custom NW reports
Disclaimer: I'm not interested in your scripts - I have my own
It seems as folks over here are thinking of charging backups so I need to provide nice overview per service. Getting out data is not a problem as I have clear naming convention which gives me overview per service and I've been exporting this data per daily basis anyway. I wish to create database out of it for nice searchable queries and graphs which also should not be a problem, but...
I have around 40k save sets per day. That's around 40k lines per day so needless to say this is million of records only after 25 days. Did I say I have data for past 2.5 years? Anyway, we started to play with this and made initial front-end in jquery, created sort of database, but obviously number of records is a problem (previously I thought of QView, but had to drop that too). I'm wondering if anyone of you has done something like this and in what tool (not how you did it - just what tool). So, in essence I'm looking for good DB model to import data and post-process it with filters and queries. Obviously we are talking about millions and millions of records here and I'm looking for some sort of open source solution
Anyone?
bingo.1
2.4K Posts
0
January 23rd, 2014 07:00
Hi Hrvoje,
as we run the production NW server on windows i use mminfo to create a monthly backup report in the first step.
Then i prefer PowerShell do all other steps like overall and client statistics.
ble1
6 Operator
•
14.4K Posts
•
56.2K Points
0
January 23rd, 2014 08:00
Thanks, will look into that.
ble1
6 Operator
•
14.4K Posts
•
56.2K Points
0
January 23rd, 2014 08:00
I can use mminfo too, but I end up with 100+ mminfo queries plus I have to do mumbo jumbo if I want to check values for any time period in the past. This is why I wish to convert that into DB format which gives me nice plate for any kind of query.
crazyrov
6 Operator
•
1.3K Posts
2
January 23rd, 2014 08:00
H,
If you are looking out for a backend DB then SQLITE should server you the purpose. Best when used on Linux - Ubuntu.
bingo.1
2.4K Posts
0
January 23rd, 2014 09:00
Well, i just generate 1 monthly mminfo report and use that report as input for my PowerShell statistics. This is much faster and does only have a one time load on the media database.
However, i have to admit that i just have about 150k save sets/month.