„FederationStats” változatai közötti eltérés

Innen: Pro-M - Oktatási Üzletág - Wiki
(→‎Running the log statistics collector)
(→‎Federation usage statistics)
 
(4 közbenső módosítás, amit 2 másik szerkesztő végzett, nincs mutatva)
2. sor: 2. sor:
 
{{TODO_EN|I am a stub, please fix me!}}
 
{{TODO_EN|I am a stub, please fix me!}}
  
−
Federation visualization project
+
Federation visualization project - discountinued.
 
* source (ruby on rails) https://repo.niif.hu/gitweb/gitweb.cgi?p=federation-stats.git;a=summary
 
* source (ruby on rails) https://repo.niif.hu/gitweb/gitweb.cgi?p=federation-stats.git;a=summary
−
* live demo https://papigw.aai.niif.hu/fedstats
+
* live demo  
  
 
== Running the sample project ==
 
== Running the sample project ==
69. sor: 69. sor:
  
 
#Config section
 
#Config section
−
PARSER_COMMAND="/usr/local/shibboleth-vho/bin/audit_r7.py"
+
PARSER_COMMAND="/opt/shibboleth-idp/bin/audit_r7.py"
−
SOURCEDIR="/usr/local/shibboleth-vho/logs"
+
SOURCEDIR="/opt/shibboleth-idp/logs"
 
TARGETDIR="/tmp"
 
TARGETDIR="/tmp"
  
−
ENTITYID="idp-entity-id""
+
ENTITYID="idp-entity-id"
 
APIKEY="aaa..."
 
APIKEY="aaa..."
 
LOCATION2PUT="https://fedstats.example.org/import_stats"
 
LOCATION2PUT="https://fedstats.example.org/import_stats"
  
−
#Should not edit below this
 
 
DATE=`date -d "yesterday" +"%Y-%m-%d"`
 
DATE=`date -d "yesterday" +"%Y-%m-%d"`
 
SOURCEFILE="$SOURCEDIR/idp-audit-$DATE.log"
 
SOURCEFILE="$SOURCEDIR/idp-audit-$DATE.log"
 +
 +
#Should not edit below this
  
 
if [ -f $SOURCEFILE ]
 
if [ -f $SOURCEFILE ]
107. sor: 108. sor:
 
fi
 
fi
  
 +
</source>
 +
 +
The script below can be used the collect statistics from all the idp audit logs placed in a folder.
 +
 +
<source lang="bash">
 +
#!/bin/bash
 +
 +
#Config section
 +
PARSER_COMMAND="/opt/shibboleth-idp/bin/audit_r7.py"
 +
SOURCEDIR="/opt/shibboleth-idp/logs"
 +
TARGETDIR="/tmp"
 +
 +
ENTITYID="idp-entity-id"
 +
APIKEY="aaa..."
 +
LOCATION2PUT="https://fedstats.example.org/import_stats"
 +
 +
FILES="idp-audit-*.log"
 +
 +
#Should not edit below this
 +
cd $SOURCEDIR
 +
for f in $FILES
 +
do
 +
  if [ -f $f ]
 +
  then
 +
    echo "Processing $f file..."
 +
    DATE=${f:10:10}
 +
    LOGINS=`$PARSER_COMMAND -l $f`
 +
    UNIQUE_LOGINS=`$PARSER_COMMAND -u $f`
 +
    SERVICES=`$PARSER_COMMAND -p $f | sed '/^[0-9]/p' -n`
 +
 +
    TARGETFILE="stat-$DATE.log"
 +
 +
    echo "ENTITYID $ENTITYID
 +
APIKEY $APIKEY
 +
DATE $DATE
 +
 +
STAT AUTH
 +
$LOGINS
 +
 +
STAT USER_COUNT
 +
$UNIQUE_LOGINS
 +
 +
STAT SSO_TO_SERVICE
 +
$SERVICES
 +
" > $TARGETDIR/$TARGETFILE
 +
 +
    wget -q --no-check-certificate --post-file=$TARGETDIR/$TARGETFILE $LOCATION2PUT -O /dev/null
 +
    rm $TARGETDIR/$TARGETFILE
 +
  fi
 +
done
 
</source>
 
</source>
  
116. sor: 167. sor:
  
 
== Creating IdPs ==
 
== Creating IdPs ==
−
You must create records for the IdPs from which the application accepts statistics. This could be done via the <code>/entities/new</code> URL. The name of the entity must be the entityID, the type must be 'idp'.
+
Use the rails console to create new idps:
  
−
=== API keys ===
+
$ RAILS_ENV=production script/console
−
Every IdP record in the database will contain a 40 character length API key.
+
 +
>> Entity.create :name => 'foo', :entity_type => 'idp'
 +
 +
=> #<Entity id: 1, name: "foo", entity_type: "idp", created_at: "2010-11-29 14:55:40", updated_at: "2010-11-29 14:55:40", api_key: "da9l233a45698fa5c4a252e301e3da2sf5ece24e">
  
 
[[Category:Csonkok]]
 
[[Category:Csonkok]]

A lap jelenlegi, 2022. október 24., 14:27-kori változata

Federation usage statistics


UnderConstruction.png TODO
I am a stub, please fix me!

Federation visualization project - discountinued.

Running the sample project

  • Install Ruby
  • Install Rails (gem install rails)
  • Setup a development.sqlite3 database with the rake db:setup command
  • Fire up script/server, it will run the project on localhost:3000

Statistic types

Currently we have the following types of statistics:

  • Unique users per day (USER_COUNT)
  • AuthnResponse per day (AUTH)
  • AuthnResponse per service per day (SSO_TO_SERVICE)

Log statistics format

The following simple format is used to convey statistics from IdPs to the central module - the white spaces (new lines) are important: 

ENTITYID #ENTITYID#
APIKEY #API_KEY#
DATE yyyy-mm-dd

STAT #STAT_ID#
xxxx

STAT #STAT_ID#
yyyy

STAT #STAT_ID#
ww | #PEER_ENTITY_1#
zz | #PEER_ENTITY_2#

The following sample might help understanding the format: 

ENTITYID https://idp.niif.hu/idp/shibboleth
APIKEY 0123.......
DATE 2009-03-18

STAT AUTH
68 logins

STAT USER_COUNT
16 unique userids

STAT SSO_TO_SERVICE
1        | urn:geant:niif.hu:niifi:sp:register.ca.niif.hu
12       | https://repo.niif.hu/shibboleth
1        | https://sandbox.aai.niif.hu/shibboleth
5        | https://sysmonitor.hbone.hu/shibboleth
10       | https://www.ki.iif.hu/shibboleth
1        | https://noc6.vh.hbone.hu/shibboleth
21       | https://webadmin.iif.hu/shibboleth
3        | https://rrd-ma.perfsonar.vh.hbone.hu/shibboleth
7        | https://ugyeletes.vh.hbone.hu/shibboleth
2        | https://noc.grid.niif.hu/shibboleth
1        | https://wiki.voip.niif.hu/shibboleth
2        | https://netmonitor.hbone.hu/shibboleth
2        | https://idp.sch.bme.hu:443/opensso/sp/test

Running the log statistics collector

This following script can be used the collect statistics from the idp audit logs of Shibboleth 2 IdP generated on the day before running. It is based on Peter Schober's audit_r7.py, and good for run from daily cronjob: 

#!/bin/bash

#Config section
PARSER_COMMAND="/opt/shibboleth-idp/bin/audit_r7.py"
SOURCEDIR="/opt/shibboleth-idp/logs"
TARGETDIR="/tmp"

ENTITYID="idp-entity-id"
APIKEY="aaa..."
LOCATION2PUT="https://fedstats.example.org/import_stats"

DATE=`date -d "yesterday" +"%Y-%m-%d"`
SOURCEFILE="$SOURCEDIR/idp-audit-$DATE.log"

#Should not edit below this

if [ -f $SOURCEFILE ]
then
    LOGINS=`$PARSER_COMMAND -l $SOURCEFILE`
    UNIQUE_LOGINS=`$PARSER_COMMAND -u $SOURCEFILE`
    SERVICES=`$PARSER_COMMAND -p $SOURCEFILE | sed '/^[0-9]/p' -n`

    TARGETFILE="stat-$DATE.log"

echo "ENTITYID $ENTITYID
APIKEY $APIKEY
DATE $DATE

STAT AUTH
$LOGINS

STAT USER_COUNT
$UNIQUE_LOGINS

STAT SSO_TO_SERVICE
$SERVICES
" > $TARGETDIR/$TARGETFILE

    wget -q --no-check-certificate --post-file=$TARGETDIR/$TARGETFILE $LOCATION2PUT -O /dev/null
    rm $TARGETDIR/$TARGETFILE
fi

The script below can be used the collect statistics from all the idp audit logs placed in a folder. 

#!/bin/bash

#Config section
PARSER_COMMAND="/opt/shibboleth-idp/bin/audit_r7.py"
SOURCEDIR="/opt/shibboleth-idp/logs"
TARGETDIR="/tmp"

ENTITYID="idp-entity-id"
APIKEY="aaa..."
LOCATION2PUT="https://fedstats.example.org/import_stats"

FILES="idp-audit-*.log"

#Should not edit below this
cd $SOURCEDIR
for f in $FILES
do
  if [ -f $f ]
  then
    echo "Processing $f file..."
    DATE=${f:10:10}
    LOGINS=`$PARSER_COMMAND -l $f`
    UNIQUE_LOGINS=`$PARSER_COMMAND -u $f`
    SERVICES=`$PARSER_COMMAND -p $f | sed '/^[0-9]/p' -n`

    TARGETFILE="stat-$DATE.log"

    echo "ENTITYID $ENTITYID
APIKEY $APIKEY
DATE $DATE

STAT AUTH
$LOGINS

STAT USER_COUNT
$UNIQUE_LOGINS

STAT SSO_TO_SERVICE
$SERVICES
" > $TARGETDIR/$TARGETFILE

    wget -q --no-check-certificate --post-file=$TARGETDIR/$TARGETFILE $LOCATION2PUT -O /dev/null
    rm $TARGETDIR/$TARGETFILE
  fi
done

Feeding the database with the statistics

The federation statistics rails project contains the script/stat_parser/file.rb command which can process the statistics format and load the data to the database. Note that this script currently contains an absolute path for the script/runner script, so you must fix this before use.

Using HTTP-Post to feed the database

When deployed, the rails project provides a /import_stats URL to which one could POST the generated statistics file.

Creating IdPs

Use the rails console to create new idps: 

$ RAILS_ENV=production script/console

>> Entity.create :name => 'foo', :entity_type => 'idp'

=> #<Entity id: 1, name: "foo", entity_type: "idp", created_at: "2010-11-29 14:55:40", updated_at: "2010-11-29 14:55:40", api_key: "da9l233a45698fa5c4a252e301e3da2sf5ece24e">
A lap eredeti címe: „https://wiki.niif.hu/index.php?title=FederationStats&oldid=4946”