20090112 DataFromVuS58

From Seamonster
Jump to: navigation, search

Heavner 14:06, 28 July 2010 (PDT) Now I don't see any data from the Campbell. Looks like 58 can't see /dev/ttyUSB0 anymore. Needs some work.

we are getting big log files, but there is a bit of confusion (on my part at least) on both the Agent log and data both going into the same log file. It is high time to look into this and sort it out.

I'm looking through the file nsrl1:~seamonster/brick58_data_Rob_pull_11_25_2008> less brick55_200810211200.log

and I see what looks like Agent logs from 2008/10/21, 2008/11/6, 2008/11/7 and 2008/11/11

2008-10-21 12: 2:38, 55, 255, 17.084999, 1, 132
2008-10-21 12: 5: 0, 55, 255, 17.084999, 1, 132

Date, Time, Brick_Number, ?, Voltage, ?, ?

Then there are many repeats of what looks like data from the Campbell from 2008/11/11

2008-11-11 13:0,3,11.88,0.9260001,90.70001,0.789,161.1,0

2008-11-11 12:45,3,11.82,0.8100001,89.1,0.413,0.144,0

2008-11-11 12:30,3,11.81,0.7330001,90.6,0.9120001,144.8,0

2008-11-11 12:15,3,11.82,0.568,90.6,0.934,158.3,0

2008-11-11 13:0,3,11.88,0.9260001,90.70001,0.789,161.1,0

2008-11-11 12:45,3,11.82,0.8100001,89.1,0.413,0.144,0

(yes, backwards temporally, but repeated MANY, MANY times... (this is

dev:~seamonster/brick58_data_Rob_pull_11_25_2008> wc brick55_200810211200.log 
wc: brick55_200810211200.log:43426: Invalid or incomplete multibyte or wide character
wc: brick55_200810211200.log:49701: Invalid or incomplete multibyte or wide character
.....
2106476  2122477 55579355 brick55_200810211200.log
dev:~seamonster/brick58_data_Rob_pull_11_25_2008> uniq brick55_200810211200.log | wc
.... (same errors as above)
2106474 2122477 55579353

Now it is 2009/03/12 taking a stab again:

root@brick58:root# crontab -l
1 */4 * * * /root/brickmove/bin/uS_data_router.py >/dev/null 2>&1
0 */1 * * * /root/brickmove/bin/housekeeping.sh >/dev/null 2>&1

#*/2 * * * * /root/brickmove/bin/uS_data_router.py >/dev/null 2>&1
#*/1 * * * * /root/brickmove/bin/housekeeping.sh >/dev/null 2>&1



#!/bin/bash

if [ -f /tmp/brickmove.pid ]
then
  kill `cat /tmp/brickmove.pid`
fi

mv /root/brickmove/logs/*.log /root/brickmove/inbound/

cat /dev/ttyUSB0 >> /root/brickmove/logs/brick55_`date +%Y%m%d%H%M`.log &
echo $! >/tmp/brickmove.pid
housekeeping.sh (END) 

#!/usr/bin/python

"""
Author: Logan Berner, (berner.logan@gmail.com), University of Alaska Southeast
This program is designed to route data from field microservers (uS)
back to the primary SEAMONSTER server. The program works in the following manner:
A function has been defined for each uS and when the program is placed on a machine
it will determine which function to call based on parsing its IP address. The program first
looks to see if there are 'log' files in the inbound directory and, if so, selects the file from
the earliest date and appends the other files in the directory to that file. All files except
for the appended file are then moved to the 'expired' directory. After clearing the 'inbound'
directory of all but one file the program attempts to ping the SEAMONSTER server. If the
ping is successful the file in the 'inbound' directory is sent via rsync and then moved to the
'expired' bin. If unsuccessful, the uS attempts to ping the uS next closest to the server and
if successful will rsync the data there instead. 
"""

from os import *
from socket import *
from time import *

inbound = '/root/brickmove/inbound/'
expired = ' /root/brickmove/expired/'
brick = 'root@192.168.1.'
ping = 'ping -c1 192.168.1.'
SM_home = 'seamonster@137.229.208.16:/home/seamonster/server_brickmove/inbound/'
inbound_files = listdir(inbound)
rsync_return = []

'''
Each function will first attempt to ping the SEAMONSTER server and if
successful will attempt to rsync the files to the server. If the ping is unsuccessful
the script will wait for 60 seconds and then attempt the ping once more. This will
continue till the ping is successful, or till it has been attempted five time. If the
rsync issuccessful the files will be moved from 'inbound' to 'expired'. If the initial
ping is unsuccessful then the uS will attempt to ping the next machine closest to
the main server and if successful rsync the files there.
'''

'''
Define Rsync Functions
'''

def rsync_SM():
    rsync_exit = system('rsync -rav --timeout=60 ' + inbound + '*.log ' + SM_home)
    if rsync_exit == 0:
        system('mv '+ inbound + '*.log' + expired)
    if rsync_exit != 0:
        rsync_return.append(1)

def rsync_uS_53():
    rsync_exit = system('rsync -rav --timeout=60 ' + inbound + '*.log ' + brick + '53:' + inboun
d)
    if rsync_exit == 0:
        system('mv '+ inbound + '*.log' + expired)    
    if rsync_exit != 0:
        rsync_return.append(1)
    
'''
Define uS Functions
'''

def uS_53():
    count = 0
    ping_exit = 99
    while ping_exit != 0 and count < 4:
        ping_exit = system('ping -c1 137.229.208.16')
        count = count + 1
        if ping_exit != 0:
            sleep(60)
        if ping_exit == 0:
            rsync_SM()
            if 1 in rsync_return:
                ping_exit = 99
                rsync_return.remove(1)
        
def uS_54():
    count = 0
    ping_exit = 99
    while ping_exit != 0 and count < 4:
        ping_exit = system('ping -c1 137.229.208.16')
        count = count + 1
        if ping_exit != 0 and count < 3:
            sleep(60)
        if ping_exit == 0:
            rsync_SM()
        if 1 in rsync_return:
            rsync_return.remove(1)
            ping_exit = 99 
        if count == 4:
            rsync_return.append(1)
    if 1 in rsync_return:
        rsync_return.remove(1)
        count = 0
        ping_exit = 99
        while ping_exit != 0 and count < 4:
            ping_exit = system(ping + '53')
            count = count + 1
            if ping_exit != 0 and count < 3:
                sleep(60)
            if ping_exit == 0:
                rsync_uS_53()
                if 1 in rsync_return:
                    rsync_return.remove(1)
                    ping_exit = 99


def uS_55():
    count = 0
    ping_exit = 99
    while ping_exit != 0 and count < 4:
        ping_exit = system('ping -c1 137.229.208.16')
        count = count + 1
        if ping_exit != 0 and count < 3:
            sleep(60)
        if ping_exit == 0:
            rsync_SM()
        if 1 in rsync_return:
            rsync_return.remove(1)
            ping_exit = 99 
        if count == 4:
            rsync_return.append(1)
    if 1 in rsync_return:
        rsync_return.remove(1)
        count = 0
        ping_exit = 99
        while ping_exit != 0 and count < 4:
            ping_exit = system(ping + '53')
            count = count + 1
            if ping_exit != 0 and count < 3:
                sleep(60)
            if ping_exit == 0:
                rsync_uS_53()
                if 1 in rsync_return:
                    rsync_return.remove(1)
                    ping_exit = 99

'''
The program checks to see that there are 'log' files before proceeding.
After confirming that 'log' files are present, the program sorts the log files,
determines the file with the earliest date, then appends the other files to
the earliest file. The non-appended files are then moved to 'expired'.
The uS IP address is then ascertained and the last two digits are used to
determine which function to call. 
'''

inbound_suffix = []
for a in inbound_files:
    inbound_suffix.append(a[-3:])       
if 'log' in inbound_suffix:   
    inbound_files.sort()
    dest = inbound_files[0]
    src = inbound_files[1:]
    for q in src:
        system('cat ' + inbound + q + ' >> ' + inbound + dest)
        system('mv ' + inbound + q + ' ' + expired)
 
    uS_ip = gethostbyname(gethostname())
    uS_tag = int(uS_ip[-2:])

    if uS_tag == 53:
        uS_53()   
    elif uS_tag == 54:
        uS_54()
    elif uS_tag == 55:
        uS_55()

root@brick58:root# cd brickmove/
root@brick58:brickmove# ls inbound/
brick55_200903120800.log
root@brick58:brickmove# cat inbound/brick55_200903120800.log 
2009-3-12  8: 2:38, 58, 187, 12.529000, 1, 132
2009-3-12  8: 4:25, 58, 188, 12.596000, 1, 132
2009-3-12  8: 6:12, 58, 186, 12.462000, 1, 132
2009-3-12  8: 7:59, 58, 188, 12.596000, 1, 132
2009-3-12  8: 9:46, 58, 185, 12.395000, 1, 132
2009-3-12  8:57: 1, 58, 190, 12.730000, 0, 132
2009-3-12  8:58:48, 58, 187, 12.529000, 0, 132
2009-3-12  9: 0:34, 58, 189, 12.663000, 0, 132
2009-3-12  9: 2:21, 58, 190, 12.730000, 0, 132
2009-3-12  9: 4: 8, 58, 190, 12.730000, 0, 132
2009-3-12  9: 5:55, 58, 185, 12.395000, 0, 132
2009-3-12  9: 7:41, 58, 187, 12.529000, 0, 132
2009-3-12  9: 9:28, 58, 190, 12.730000, 0, 132
2009-3-12  9:11:15, 58, 187, 12.529000, 0, 132
2009-3-12  9:56:50, 58, 190, 12.730000, 0, 132
2009-3-12  9:58:37, 58, 187, 12.529000, 0, 132
2009-3-12 10: 0:24, 58, 188, 12.596000, 0, 132
2009-3-12 10: 2:10, 58, 188, 12.596000, 0, 132
2009-3-12 10: 3:57, 58, 186, 12.462000, 0, 132
2009-3-12 10: 5:44, 58, 190, 12.730000, 0, 132
2009-3-12 10: 7:30, 58, 189, 12.663000, 0, 132
2009-3-12 10: 9:17, 58, 186, 12.462000, 0, 132
2009-3-12 10:11: 5, 58, 187, 12.529000, 0, 132
2009-3-12 10:56:40, 58, 188, 12.596000, 0, 132
2009-3-12 10:58:27, 58, 187, 12.529000, 0, 132
2009-3-12 11: 0:13, 58, 190, 12.730000, 0, 132
2009-3-12 11: 2: 0, 58, 189, 12.663000, 0, 132
2009-3-12 11: 3:47, 58, 186, 12.462000, 0, 132
2009-3-12 11: 5:33, 58, 189, 12.663000, 0, 132
2009-3-12 11: 7:20, 58, 187, 12.529000, 0, 132
2009-3-12 11: 9: 7, 58, 186, 12.462000, 0, 132
2009-3-12 11:10:53, 58, 187, 12.529000, 0, 132
2009-3-12 11:52:49, 58, 185, 12.395000, 1, 132
2009-3-12 11:54:37, 58, 186, 12.462000, 1, 132
2009-3-12 11:56:24, 58, 185, 12.395000, 1, 132
2009-3-12 11:57:27, 58, 189, 12.663000, 0, 132
2009-3-12 11:58:11, 58, 186, 12.462000, 1, 132
2009-3-12 11:59:19, 58, 186, 12.462000, 1, 132
2009-3-12 11:59:57, 58, 185, 12.395000, 1, 132

root@brick58:root# ps auxww
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.8  1368  512 ?        S    11:46   0:01 init [2]  
root         2  0.0  0.0     0    0 ?        S    11:46   0:00 [keventd]
root         3  0.0  0.0     0    0 ?        SN   11:46   0:00 [ksoftirqd_CPU0]
root         4  0.0  0.0     0    0 ?        S    11:46   0:00 [kswapd]
root         5  0.0  0.0     0    0 ?        S    11:46   0:00 [bdflush]
root         6  2.4  0.0     0    0 ?        S    11:46   0:28 [kupdated]
root         7  0.0  0.0     0    0 ?        S    11:46   0:00 [mtdblockd]
root        29  0.0  2.7  2584 1676 ?        Ss   11:47   0:00 /sbin/devfsd /dev
root       121  0.0  0.0     0    0 ?        S    11:47   0:00 [khubd]
daemon     383  0.0  0.7  1532  452 ?        Ss   11:47   0:00 /sbin/portmap
root       412  0.6  4.0  4704 2496 ?        Ss   11:47   0:07 /usr/sbin/openvpn --writepid /var/run/openvpn.client.pid --daemon ovpn-client --status /var/run/openvpn.client.status 10 --cd /etc/openvpn --config /etc/openvpn/client.conf
root       417  0.0  0.8  1428  532 ?        Ss   11:47   0:00 /usr/sbin/inetd
root       430  0.0  2.7  4924 1660 ?        Ss   11:47   0:00 /usr/sbin/sshd
root       435  0.0  1.2  1624  748 ?        Ss   11:47   0:00 /sbin/rpc.statd
root       440  0.0  1.3  1876  848 ?        Ss   11:47   0:00 /usr/sbin/cron
root       488  0.0  8.2  7156 5040 ?        Ss   11:47   0:01 /usr/sbin/munin-node
root       495  0.0  0.8  1372  508 ?        S    11:47   0:00 /root/bin/Agent 1 1 132 10 0 4 2 2 10
root       502  0.0  0.8  1360  496 vc/1     Ss+  11:47   0:00 /sbin/getty 38400 tty1
root       503  0.0  0.8  1360  508 ttyAM0   Ss+  11:47   0:00 /sbin/getty -L ttyAM0 115200 vt100
root       504  0.0  0.8  1360  508 ttyAM1   Ss+  11:47   0:00 /sbin/getty -L ttyAM1 115200 vt100
root      1133  0.5  3.9  5896 2400 ?        Ss   11:52   0:04 sshd: root@pts/0 
root      1134  0.1  2.4  2620 1480 pts/0    Ss   11:52   0:00 -bash
root      1844  0.0  0.5  1356  352 ?        S    12:00   0:00 cat /dev/ttyUSB0
root      2757  0.0  1.4  2572  860 pts/0    R+   12:06   0:00 ps auxww

root@brick58:root# dmesg
Linux version 2.4.26-ts11 (eddie@io) (gcc version 3.3.4) #1 Tue Nov 14 13:55:04 MST 2006
CPU: Arm920Tid(wb) revision 0
Machine: ep9301
On node 0 totalpages: 2048
zone(0): 2048 pages.
zone(1): 0 pages.
zone(2): 0 pages.
On node 1 totalpages: 2048
zone(0): 2048 pages.
zone(1): 0 pages.
zone(2): 0 pages.
On node 5 totalpages: 2048
zone(0): 2048 pages.
zone(1): 0 pages.
zone(2): 0 pages.
On node 16 totalpages: 2048
zone(0): 2048 pages.
zone(1): 0 pages.
zone(2): 0 pages.
On node 17 totalpages: 2048
zone(0): 2048 pages.
zone(1): 0 pages.
zone(2): 0 pages.
On node 18 totalpages: 2048
zone(0): 2048 pages.
zone(1): 0 pages.
zone(2): 0 pages.
On node 19 totalpages: 2048
zone(0): 2048 pages.
zone(1): 0 pages.
zone(2): 0 pages.
Kernel command line: console=ttyAMO,115200 root=/dev/ram0 init=/linuxrc rw
Relocating machine vectors to 0xffff0000
Console: colour dummy device 80x30
Calibrating delay loop... 99.94 BogoMIPS
Memory: 8MB 8MB 8MB 8MB 8MB 8MB 8MB 8MB = 64MB total
Memory: 60580KB available (1208K code, 336K data, 72K init)
Dentry cache hash table entries: 8192 (order: 4, 65536 bytes)
Inode cache hash table entries: 4096 (order: 3, 32768 bytes)
Mount cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer cache hash table entries: 4096 (order: 2, 16384 bytes)
Page-cache hash table entries: 16384 (order: 4, 65536 bytes)
CPU: Testing write buffer: pass
POSIX conformance testing by UNIFIX
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
devfs: v1.12c (20020818) Richard Gooch (rgooch@atnf.csiro.au)
devfs: boot_options: 0x1
ttyAM0 at MMIO 0x808c0000 (irq = 52) is a AMBA
ttyAM1 at MMIO 0x808d0000 (irq = 54) is a AMBA
ttyAM2 at MMIO 0x808e0000 (irq = 55) is a AMBA
pty: 1024 Unix98 ptys configured
Serial driver version 5.05c (2001-07-08) with MANY_PORTS SHARE_IRQ enabled
Real Time Clock Driver v1.10f
ep93xx_eth() version: ep93xx_eth.c: V1.0 09/04/2003 Cirrus Logic
RAMDISK driver initialized: 16 RAM disks of 12288K size 1024 blocksize
Searching for NAND flash...
NAND device: Manufacturer ID: 0xec, Chip ID: 0x75 (Samsung NAND 32MiB 3,3V 8-bit)
Scanning device for bad blocks
Using static partition definition
Creating 3 MTD partitions on "NAND 32MiB 3,3V 8-bit":
0x00000000-0x00004000 : "TS-BOOTROM"
0x00004000-0x01d04000 : "Linux"
0x01d04000-0x02000000 : "RedBoot"
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP: Hash tables configured (established 4096 bind 8192)
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
NetWinder Floating Point Emulator V0.97 (double precision)
RAMDISK: Compressed image found at block 0
RAMDISK: ran out of compressed data
length error<6>Freeing initrd memory: 438K
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
VFS: Mounted root (ext2 filesystem).
Mounted devfs on /dev
Freeing init memory: 72K
Warning: unable to open an initial console.
sdcard0: Technologic Systems SD card controller, address 0x13000000
sdcard0: card size 3970048 sectors
Partition check:
 sdcard0a: p1 p2 p3
usb.c: registered new driver usbdevfs
usb.c: registered new driver hub
zd1211 - http://zd1211.ath.cx/ - r83
Based on www.zydas.com.tw driver version 2.5.0.0
usb.c: registered new driver zd1211
usb-ohci.c: USB OHCI at membase 0xe0020000, IRQ 56
usb.c: new USB bus registered, assigned bus number 1
hub.c: USB hub found
hub.c: 3 ports detected
Universal TUN/TAP device driver 1.5 (C)1999-2002 Maxim Krasnyansky
hub.c: new USB device not_pci-1, assigned address 2
usb.c: USB device 2 (vend/prod 0x67b/0x2303) is not claimed by any active driver.
usb.c: registered new driver serial
usbserial.c: USB Serial support registered for Generic
usbserial.c: USB Serial Driver core v1.4
usbserial.c: USB Serial support registered for PL-2303
usbserial.c: PL-2303 converter detected
usbserial.c: PL-2303 converter now attached to ttyUSB0 (or usb/tts/0 for devfs)
pl2303.c: Prolific PL2303 USB to serial adaptor driver v0.10
root@brick58:root# 

OK, quick conclusions.. the campbell may be plugged into the prolific USB/serial? (I can't get an answer on /dev/ttyS0 or dev/ttyS1). So I'm trying "cat /dev/ttyUSB0" and leaving it listening for over a minute to see if I can get some campbell data.

Nope.. One minute not enough.

I killed the Agent on VuS so I can play a bit more..

OK, housekeeping.sh starts listening to the USB device. Is the campbell plugged into it?


2009/03/23 Aha! The campbell battery had died, so no campbell.

Now, with a new battery (thanks Nick!), the campbell is spitting out data, 4 points a second. So housekeeping.sh is trying to listen to /dev/ttyUSB0 and is getting about (20 min) * (60 sec/min) * (4 data lines / secon) for all of 5 unique data points (the campbell gets one new data line during the approx. 20 minute the VuS is up).

So for the short term, I rsynced brick58/VuS:~brickmove/expired/* to seamonster@nsrl1:~/brick58_afewfrom_20090323/

Then used these two pieces of magic:

#!/usr/bin/perl
# Matt Heavner, 2009/03/23
# The VuS program is doing a:
# cat /dev/ttyUSB0 > brickXX_ymdhms.log &
# However, the brick is spitting out 4 values every second, so we are getting
# lots, and lots, and LOTS of duplicate data..  (about 20 min x 60 sec..)
# But at some point the data does get a new point, just to be paranoid
#  and not lose data..

# PLAN:
# sort $file | uniq > $file_uniq

opendir(DIR,".");
@files = grep(/brick55/,readdir(DIR));
closedir(DIR);
foreach $file (@files) {
    `sort $file | uniq >> fnice`;
}

open(FNICE,"fnice");
foreach $line (<FNICE>) {
    if ($line =~ /^(\d\d\d\d-)(\d-.*)/ ) { $line = "$1"."0$2\n";}
    if ($line =~ /^(\d\d\d\d-\d\d-)(\d\s.*)/ ) { $line = "$1"."0$2\n";}
    if ($line =~ /^(\d\d\d\d-\d\d-\d\d\s)(\d\072.*)/ ) { $line = "$1"."0$2\n";}
    if ($line =~ /^(\d\d\d\d-\d\d-\d\d\s\d\d\072)(\d\054.*)/ ) { $line = "$1"."0$2\n";}
    if ($line =~ /^\d\d\d\d-\d\d-\d\d\s\d\d\072\d\d/ ) { print $line;}
}

and

eamonster@dev:~/brick58_afewfrom_20090323$ ../mjh_sort_VuS_dups.pl > ../MM_Clean_wx
seamonster@dev:~/brick58_afewfrom_20090323$ sort /home/seamonster/MM_Clean_wx > ~/Vus58_MM_20080323_clean.txt

There was still some Brick data (in addition to the Campbell data) in there.

And I see why some files are brick55 and some brick58! Housekeeping is hard wired to 55... Also need to look at the data transfer script.. no 53 is around, so I think that's why it isn't auto-cleaning properly.

2009/04/02

At 8am, log in, rsync -razv brickmove/expired seamonster@10.8.1.1:58/ (this has both brick55_ and brick_58). The "brick_55" is housekeeping (and includes the campbell data), "brick_58" is microcontroller/Agent logs (voltage, "agent" state, etc..)

got all of expired moved and deleted.. now starting inbound.. inbound has brick55.. (starting w/ 20090312)--only three files..

inbound/brick55_200903120800.log
inbound/brick58_200904020802.log
inbound/brick58_200904020804.log

got 'em all, removed 'em, now disk is back to 70%

2009/04/22

JoshG 10:58, 22 April 2009 (PDT) This is interesting, I rsynced both of these files to my folder on nsrl from 58. The first log file I got at 4pm yesterday, and contains the four values (:00, :15, :30, :45) from the hour prior repeated over and over. The second file contains the four values from the 7am hour this morning, but with a LOT more repetitions.

-rw-r--r--  1 josh josh  16K 2009-04-21 16:01 brick58_200904211600.log
-rw-r--r--  1 josh josh 135K 2009-04-22 08:10 brick58_200904220800.log