Advice for Integrating Satellites

After all, we thought we were smart students and didn't need all the overbearing documentation that makes industry satellites so slow and expensive (Dinosaurs! Dinosaurs!). We didn't need other people telling us how to run our project. Here's where we could have used their advice: 

Recommendation	Our "Lesson"
Get Spares of Everything	This is a general rule; I've never known our project to have "too many" of anything. Extra tools always come in handy if you want several people to dismantle the side panels. Extra fasteners mean you're not scrounging around. Extra L-brackets means that when one breaks (see below), then you're not worried that you're going to have to make more.
Check every fastener and hole for fit and proper size/number of items	We had to build a second set of side panels for the shake test, because our solar panels were not finished. Unfortunately, they were a bit of a rush job and the holes did not line up exactly. I personally stripped and popped out a good half-dozen pem nuts from our flight structure in my attempts to tighten down the panels. When it's 9pm and the shake test is at 9am, and you start breaking flight hardware, let me tell you, the lab is not a fun place to be. The second happy story concerns our CPU box, which had to be filed out a bit because its fasteners couldn't line up with the tray. Of course, we caught that problem at 7pm...
Drill all holes at the same time - especially if they're supposed to line up	This traumatic event happened to our solar cells, which were bonded to the panels and soldered together before we noticed that one of the holes was missing. On top of that, none of the holes were aligned with the top honeycomb panel - they were all fractions of an inch off. Two students, two files, one afternoon. Not a lot of fun.
Test the expected telemetry values - do this as soon as the electronics are wired up	Four months after establishing the telemetry list and reading the values, somebody finally realized that the THD sensors (our main payload) were misnumbered in their outputs. (Why? Well, the sensors don't do much that's interesting in the lab, so we didn't pay close attention to the signals. Moreover, we hadn't sat down with the PI and defined the expected telemetry output.) Secondly, we realized six months into the functional flight hardware tests that nobody had checked some of the temperature sensors (they hadn't been integrated - though the signal conditioning electronics were on board). A miswiring had damaged one of the boards and we didn't catch it until after shake. (That has since been fixed, but it was a bit of a panic trying to figure out what went wrong.) What this means is that even if the proper hardware is not in place, wire together some sort of kluge to verify that what you DO have is working properly. That way, you can keep an eye on it and save trouble when the full integrations comes around.
Prominently post telemetry values and/or go over them with the pertinent people	This relates to how long it took for us to realize that the THDs were mistakenly labeled. In addition, the expected values are essential for troubleshooting problems, as the next example illustrates.
Start a binder with sample telemetry from every major test or reconfiguration	Not only did the THDs have the wrong outputs labeled, but somewhere between September 1995 and March 1996, one of the sensors broke. (Yes, this is our primary payload. Thanks for bringing that up.) We had gone through three thermal cycles and two vacuum test in that time, but hadn't really checked the telemetry for long-term variations. (Why? Because, again, we hadn't sat down with the PI to get expected values.) And when we had taken data, we hadn't kept it, which means now we don't know when/how the sensor broke, so we'll have to do another vacuum test to make sure the sensors will work.
Double check interior space before adding components	A temperature sensor was epoxied onto the CPU in such an orientation that we couldn't close the box. We had to (carefully) scrape off the epoxy and move the sensor.
Have somebody else double-check your wiring connections	One of the wires that ran telemetry data from the Power tray to the CPU was omitted during our construction. Since we didn't have a list of expected values, and hadn't done a detailed telemetry analysis (are you detecting a theme?), this mistake wasn't caught for six months.
Confirm that the space-rated chips fit into the sockets you've created for them - this applies to all components, actually	We have had months of redesigns (and then a very time-consuming soldering job - thank you, Godwin Zhang!!!) because the space-rated chips donated to us weren't in the package that we expected.
Queue up jobs so that you can use expensive materials at one sitting.	We have used three bags of thermal epoxy for about 1/4 bag's worth of epoxying because the parts weren't ready at the same time. We've wasted probably 2/3 of our conformal coat for the same reasons. I could tell similar stories about our other epoxies and RTV.
Finish whole boxes, whole boards	For our CPU, TNC and telemetry boards, we had all but one or two chips conditioned for flight (back to getting those space-rated chips to fit). It is not an easy task to just "go back" and touch up the board once they're ready. We thought it would be; we were wrong. What happens is that you stall and wait for that piece to be done - or you let other parts of the project slide becaue you've got that piece of unfinished business. Closing up boxes and locking down trays builds excitement and momentum that is easily killed by waiting for "just one or two" parts.
Discretely group signals and connections coming off the tray	The simplest and often most effective debugging option is to check the pins coming off the connector. If you've got power and signals on the same connector with everything gooped up, it takes work to both power the component and check the output. Specifically, the temperature sensor on the Comm tray wasn't doing what was expected of it. I wanted to stick a voltmeter on the pins coming out, but power for the sensor was supplied through the same connector. I ended up having to scrape some of the RTV away to find contact points, which means I had to remember to go back and add RTV. (On our other trays, this is not an issue, since the power harness has a separate connector.)
A functional engineering model (with the same wiring), a CPU debug port, a power supply that measures current, and a scanner are all invaluable diagnostic tools	I cannot begin to describe how much time these items save, but I'll try. So many times we have swapped a questionable component onto our engineering model and been able to quickly diagnose a problem. So many times I have directly connected to the CPU to confirm whether the code was working - often it was shown to be a TNC connection problem. So many times I have been able to look at the current supplied to the satellite to confirm whether or not the camera is turning on or if the satellite has reset or (unfortunately) if there's a short. With the radio frequency scanner, we can listen in to the transmitter packets - and could have realized that we had set up our ground station incorrectly, corrected it, and continued the thermal cycle test. Instead, we went home, discovered the problem, and had to go back the next week to try again.
Build with a modular architecture concept	The ease with which I can pull out the TNC, which is acting weirdly again, and put in our engineering model's TNC, which I can already confirm is working properly, to isolate problems is a tremonedous troubleshooting tool. It saves time, every time. On top of that, we can go from separate boxes on the benchtop to the fully assembled satellite in about thirty minutes.

Lessons Learned, Part Two: Operational Experience