We are currently working on our servers, so there will be some unexpected network issues over the next half hour or so. Sorry for any inconveniences.
Temporary network issues
“webtimetables plus” project proposal
Hey Folks
Project proposal for redpill. This is will not be a teaching workshop but a chance for anyone who is interested in coding to work together on a project.
Problem: Webtimetables, it goes down often for large periods of time to backup, it crashes a lot, slow and looks god awful plus you don’t get notifications when something changes.
I came up with a few ideas on the bus on what we could do. Myself and apparently some other people have written some variously polished versions of this (mine being pretty crappy and makeshift). I want to expand on it and run it into a full blown project with a lot more to it though.
- Pull out the data giving a course code and a year.
- See if we can display this in many nice ways. For example, pdf, push it into a google docs spreadsheet using their api.
- Cache the result.
- Query webtimetables periodicly and check if any changes have been made
- publish results by (email, mailing list, google groups, rss feeds, carrier pigeon)
- Fancy gui, click the module, have all the other instances highlighted, see when it’s next one
- mobile android app (or iphone if you have a spare 100 euro for a license) where you save your course + year and it pulls the results out of google docs in the form of “what class do i have now, what class do i have next, what room etc”
- Who’s in this room – pull out timetables for rooms
- we could generate timetables as all sorts of files and drop them either in a google apps collection folder or in dropbox or give out the raw data as xml,csv etc.
- monitor when webtimetables is online/offline + graph it and tweet out alerts when it’s down/up
- command line version of all of the above
Unplanned outage
Was hoping to post about more exciting things that have happened but for the moment, our servers are down due to unplanned outage. We will post an update when we have the servers backup and running along with a technical summary.
Update: Back online and as promised an explanation.
The problem has a number of contributors mostly consisting of stupid mistakes that should not have happened but lessons well learned.
First a little background:
Main login server authenticates off an ldap server so we can do central authentication with many different services (lots of logins, one password/user data etc). SSH servers run nslcd, a daemon that queries nss details from an ldap server (such as username, uid, shell, home directory etc). NSCD (different to nslcd), is a caching daemon that caches results from the ldap server.
Factors in downtime:
1:NSCD was turned off (caching daemon). By itself wasn’t that big an issue but still not recommended. The correct response was to turn it off only while debugging or to force the cache to update when we need it to update.
2: Debugging was enabled on the ldap server to a separate non syslog file with excess debugging information. I had turned it on to debug an issue and just left it on by accident.
3: /var/log or even /var /was not in a separate directory. Not anticipating logs to grow that fast before logrotate took care of it was my fault. openldap was logging directly to a file which didn’t go through syslog, getting around that point. I imagine that even if we had separated /var, it would have still caused issues as slapd (openldap server), still stores its real files for non special databases in /var/lib/ldap.
4: Even with all of these issues, it was processes on our main login server that started the ball rolling. When we tested it ourselves, because we weren’t causing the server to lookup much, we didn’t see much but as soon as people started using it and we started getting more members, even doing a simple ls -al of the home directories, causes the server to do 100s of queries, each for specific uidNumbers due to the amount of users and then of course, each user doing similar commands etc etc, you can see how it went.
Investigation
We didn’t actually know what the problem was at the start. The possible candidates came to mind: someone was brute forcing the server, someone was scraping or rapidly querying the webserver in such a way that caused it to query the ldap server. (wordpress is inter linked with ldap), something was misconfigured in nslcd on one of the servers, someone was querying our ldap server from inside our network excessively hard.
When I saw the hard drive was full on the ldap server, I took down the server, fired up a new vm, setup a separate /var/log partition and rebuilt the server. This was so I could properly examine the logs in case anything more malicious was at hand while letting the ldap service run. Pretty easy process with slapcat however, my documentation on rebuilding was slightly ambiguous on recent changes and caused me to take longer than I thought it would.
Mitigating the problem + future proofing it.
/var/log now has it’s own separate partition. NSCD is turned on and will continue to be tweaked to get the right cache time. Logging verbosity has been reduced to a more sensible level and only uses syslog. I’ve correctly fixed/updated the documentation on rebuilding. In the case that something happens again, i’m going to write a script to build the server in one go without prompting to save some time as well . We’re also looking into better monitoring solutions that will give us rapider response times so we can react faster.
This is a little embarrassing I admit, but I’d prefer to keep you all informed about what’s actually going on. If anyone is interested and wants to know more, hit us up.
Mark
Trip to campuscon WIT form
Please fill in this form indicating whether you’d like to come down for campuscon WIT with us. We will be posting up details and locations for transport and accommodation so check back with periodically.
https://docs.google.com/spreadsheet/viewform?formkey=dEtvRjgxaHlMYjhWOFVKWFMzUGJTbHc6MQ
Here’s what we’ve got so far
Transport: Bus – buseireann public bus return ticket – approx 13 euro (cheaper if you book online)
http://www.buseireann.ie/pdf/1307702391-4.pdf
Accommodation: single/double/triple bed (all for 49 euro a night). If you can get 3 people to split it, that’s 16.3 euro per person per night. (location tinyurl.com/ramadahotellocation)
As you can see from campuscon location (tinyurl.com/campusconlocation) it’s quite close
Conference cost – 5 euro on the door.
Note: campuscon link – http://campuscon.hackingwit.com/
WIT Campuscon hacking challenge
WIT’s hacking society are putting on a full day event on 21st January where there will be a team hacking event http://campuscon.hackingwit.com/.
With some great feedback on facebook, we’ll be going down on 20th with bus eireann (12.60 euro a ticket if you book online). WITHacking are charging 5 euro into the event at the door. We’re still looking around for the best accommodation. We may also be able to subsidise part of the trip. Hope you can all come!
We want to organise a time we can all meetup to practice/train for the event and decide on teams to try our hand at the hacking. Remember, everyone’s a beginner, you’re guaranteed to have less fun and learn less if you don’t give it a try! This will be different to our regular workshops where it’ll be a meetup session where it is a group effort to practice and train rather than us preparing specific materials and you just following along.
Thursday 19th 12pm for a full day – official first day to come out and train / practice. Relaxed group orientated environment.
Requirements:
-Laptop
-Linux of some sort installed or on a cd (backtrack/ubuntu cds we handed out will do grand)
-Usb stick to install a persistent version might be handy too
-Network cable if you have one
-Power extension thinamabobs if you have them as well.
-Bring along some isos or vms of debian/ubuntu + windows if you have them for people who feel more technically comfortable.
To start off with, we’ll be focusing on web application vulns (most likely sql injection and further escalation from there) as it’s stated in http://campuscon.hackingwit.com/events.php.
If you want to get a jump start, check out http://www.hackthissite.org/ and http://sourceforge.net/projects/lampsecurity/ to begin with
Computer Security workshop on tonight! Next weeks sessions postponed
Computer security talk Monday 28th
We will be hosting a two part Computer Security talk on Monday 28th November 6pm following up with a workshop on Tuesday 29th November 6pm and a part two the following week. Please bring laptops on Tuesday if you have them.
Update! We have a room, see the bottom of the post.
Leading up to the talk, we will be posting decoding challenges on twitter. Crack the code and send us in the answer and there could be some loot awaiting you Monday.Follow @DITNetsoc
I will be diving right into the topic and show you real examples of how systems can be attacked and defended and talking about the penetration testing, a carear path where you are hired to professionally and legally, test the security of a system. We’ll be covering where to get started, books and mailing lists to follow and websites to visit.
We’ll be covering various attack methodologies and giving you a glimse of what’s inside malware along with popular tools that are often used to test security along with solutions to help you avoid becoming the next victim.
If that doesn’t sound exciting enough, we have built some systems we’ll be demoing a hack against live at the event and will be aiming to teach you how to do so and how to make sure when you’re writing code, that you don’t leave these holes open. If you were at our last PHP workshop, you will have gotten a taste of what you can possibly do. Come along to our security talk to learn how attacks are carried out and how to defend against them.
Dates and Locations
What: Computer Security Talk
When: 6pm Monday 28th November
Where: Kevin St KA-G-28
What: Computer Security Workshop
When: 6pm Tuesday 28th November
Where: Kevin St KA-1-16
What: Computer Security Talk (Part 2)
When: 6pm Monday 5th December
Where: Kevin St KA-G-28
What: Computer Security Workshop (Part 2)
When: 6pm Monday 6th December
Where: Kevin St KA-1-16
Integrated login to our website
We have integrated our ldap authentication with our website netsoc.dit.ie (this one). All current members should be able to login. Give it a try!
https://www.netsoc.dit.ie/wp-login
Note logins do require https so it is as secure as https. We do have a signed https cert however due to firefox not having some required intermediate CA certs, it may appear invalid on firefox. We are working on supplying the right CA crt chain in apache and will fix this very soon.
PHP Workshop 2!
We’ll be putting on another php workshop at 6pm KA-1-16 on Tuesday 8th November 2011. We have fixed the unfortunate technical details that cropped up last week.
Again no prior knowledge require, bring laptops if you have them and yourself.
We have changed the structure of the workshop and will now be splitting into two groups. One that have had experience with programming before, and know some basic html and those who are in first year or have very little programming experience and know little to know about html.
Look forward to seeing you all there!
Motivated Individuals – We Need YOU!
Roll up, roll up, read all about it! Netsoc are in search of two new committee members to assist in the day-to-day running of the society. If you have an interest in either of the following roles please send off an e-mail to net…@…ie, with a short description of why you think you are most suited for the position :
- Public Relations Officer
- General Committee Member
Follow Us!