JAW Speak

Reading time: 3 – 4 minutes

Web apps have numerous choices for storing stateful data in between requests. For example, when a user goes through a multi-step wizard, he or she might make choices on page one that need to be propagated to page three. That data could be stored in http session. (It also could get stored into some backend persistence and skip session altogether).

So where does session get stored? There are basically four choices here.

1. Store state on one app server, i.e. for java in HttpSession. Subsequent requests need to be pinned to that particular app server via your load balancer. If you hit another app server, you will not have that data in session.
2. Store state in session, and then replicate that session to all or many other app servers. This means you don’t need a load balancer to anchor a user to one app server; multiple requests can either hit any app server (full replication). Or use the load balancer to pin to particular cluster (themselves replicating sessions, and giving higher availability). Replication can be smart so only the deltas of binary data are multicast to the other servers.
3. Store the state on the client side, via cookies, hidden form fields, query strings, or client side storage in flash or html 5. Rails has an option to store it in cookies automatically. Consider encrypting the session data. However, some of these options can involve a lot of data going back and forth on every request, especially if it’s in a cookie and images/scripts are served from the same domain.
4. Store no state on app servers, instead write everything in between requests down to backend persistence. Do not necessarily use the concept of http session. Use id’s to look up those entities. Persistence could be a relational database, distributed/replicated key/value storage, etc. Your session data is serialized in one big object graph, or as multiple specific entries.

What you choose is up to many factors, however a few guidelines help:

1. Try to keep what is in session small.
2. If possible, keep session state on the client side.
3. Prefer key/value storage replicated among a small cluster over replicating all session state among all app servers.
4. Genuinely consider sticky sessions, which home users to a particular app server. Many benefits abound, including what Paul Hammant talks about w.r.t Servlet spec 7.7.2 Appengine’s Blind Spot.
5. If you serialize an object graph, recognize that when you do a deployment it will probably mean existing sessions are now unable to be deserialized by the newly deployed app. Avoid this by using your load balancer to swing new traffic into the new deployments, monitor errors, and then let the old sessions expire before switching all users over to the new deployment. Bleed them over.
6. Session is not for caching. It may be tempting to store data in session for caching purposes, but soon you will need a cache.
7. Store in session what is absolutely necessary for that user, but not more. See caches, above.

Reading time: 2 – 4 minutes

We are using FactoryBeans extensively to own the responsibility of interesting work in constructing our object graph. These are request scoped, because they may depend on other objects that were themselves created by FactoryBeans specific to this request. For example, we have a Customer which needs an Account, so CustomerFactoryBean depends on the field Account to be injected. (We use field or setter injection because otherwise creating a factory bean that depends on another object that is created by a factory bean may create a spring exception about circular dependencies.)

package com.example.dotcom;
 
import com.example.common.Account;
import com.example.dotcom.session.CustomSession;
import org.springframework.beans.factory.FactoryBean;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Scope;
 
// the scope is crucial, this controls the scope of the factory bean
@Scope("request")
public class AccountFactoryBean implements FactoryBean {
 
       // this is itself created from another factory bean
       @Autowired
       CustomSession session;
 
	@Override
	public Object getObject() throws Exception {
		// Sometimes this is going to return null, other times an instance.
                // The fix is to always return a non-null instance (despite what the
                // javadoc says). Use the Null Object pattern.
                return session.get(Account.class);
 
 
		// Note: you will need to implement caching in here or in 
		// a custom scope (which we did). Or every time you need
		// to inject the Account it will call getObject(), which
		// could be problematic if it was a slow operation.
	}
 
	@Override
	public Class getObjectType() {
		return Account.class;
	}
 
	@Override
	public boolean isSingleton() {
		return false;
	}
}

We are using autowiring by type, but we encountered a problem when the result of a factory bean’s getObject() is sometimes null. If the first call to that request scoped factory bean returned null, it would never call getObject() again in future requests. Upon investigation, you can see there is a subtle caching of null return values from factory beans. (Even if they are request scoped.) Might be a bug, I don’t know. But the work around is probably a good idea to implement anyways: use Null Objects instead of passing around null.

Here is the problematic code from Spring that caches the null return value: AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement. There is a field cached and cachedFieldValue. These are set if the value is null, and then on subsequent calls they will return the null value without calling getObject on the factory bean.

I recommend not returning null in factory beans, or else it might be cached by Spring and your factory bean will not be called again when that object is needed.

Reading time: 3 – 4 minutes

Today I was working with Kumar and Elian where we encountered production logs with dozens of NullPointerExceptions with no stack trace. We identified that the JIT compiler will optimize away stack traces in certain exceptions if they happen enough. The code below reproduces it. We are on jdk 1.6 (I don’t remember the minor version now).

public class NpeThief {
    public void callManyNPEInLoop() {
        for (int i = 0; i < 100000; i++) {
            try {
                ((Object)null).getClass();
            } catch (Exception e) {
                // This will switch from 2 to 0 (indicating our problem is happening)
                System.out.println(e.getStackTrace().length);
            }
        }
    }
    public static void main(String ... args) {
        NpeThief thief = new NpeThief();
        thief.callManyNPEInLoop();
    }
}

Run it as follows and the issue will appear (the stack trace length changes from 2 to 0):
javac NpeThief.java && java -classpath . NpeThief
javac NpeThief.java && java -server -classpath . NpeThief


How to fix it? The following options resolve it and it stays at 2, never going to 0 as length of NPE’s stack trace:
javac NpeThief.java && java -XX:-OmitStackTraceInFastThrow -server -classpath . NpeThief
javac NpeThief.java && java -XX:-OmitStackTraceInFastThrow -classpath . NpeThief
javac NpeThief.java && java -Xint -classpath . NpeThief
javac NpeThief.java && java -Xint -server -classpath . NpeThief

So the solution is to start with -XX:-OmitStackTraceInFastThrow argument to java which instructs the JIT to remove this optimization [1,2], or operate in interpreted only mode [3]. What is going on? [2]

The JVM flag -XX:-OmitStackTraceInFastThrow disables the performance optimization of the JVM for this use case. If this flag is set, the stacktrace will be available. (editor: or if the -Xint is set).

“The compiler in the server VM now provides correct stack backtraces for all “cold” built-in exceptions. For performance purposes, when such an exception is thrown a few times, the method may be recompiled. After recompilation, the compiler may choose a faster tactic using preallocated exceptions that do not provide a stack trace. To disable completely the use of preallocated exceptions, use this new flag: -XX:-OmitStackTraceInFastThrow.” http://java.sun.com/j2se/1.5.0/relnotes.html

How does this impact performance? I did extremely naive timing and time java [args] -classpath . NpeThief. It behaved as expected, with interpreted the slowest. Is that the solution?

No. We weren’t going to change production JVM options to resolve this, instead since our isolated example code above indicated that initial exceptions would throw with the full stack trace, we went back into older logs and grepped. Sure enough, right after we deployed there were exceptions appearing with the full stack trace. That gives us enough information where we can identify and fix the bug.

Notes:
[1] Related sun bug 4292742 NullPointerException with no stack trace
[2] Helpful StackOverflow discussion of NullPointerException missing stack traces
[3] -Xint: Operate in interpreted-only mode. Compilation to native code is disabled, and all bytecodes are executed by the interpreter. The performance benefits offered by the Java HotSpot VMs’ adaptive compiler will not be present in this mode.

Reading time: 3 – 5 minutes

We’re (stuck*) using maven on a project, and this means often the build doesn’t act like you’re expecting. Debugging requires knowledge about maven’s capabilities and less documented features. Here are some tips, please add more in the comments.

1. The -X flag shows debug logging, and tee it to a file which you then search for clues of what happened. This is great for verifying if a plugin actually ran, and what it did. Example: mvn -X install | tee out.txt
2. mvn dependency:tree (show the transitive dependencies of your dependencies) [1]
3. -Dtest=MyJavaTest (just use the name of the class, no package necessary)
4. mvn help:system (show environmental variables and system properties) [2]
5. mvn help:active-profiles (show the active profiles to be run)
6. mvn help:all-profiles (show all profiles maven reads)
7. mvn help:effective-pom (show the logical pom.xml being used)
8. mvn help:effective-settings (show the logical settings.xml which is being used)
9. mvn help:describe -DgroupId=org.apache.maven.plugins -DartifactId=maven-resources-plugin -Ddetail (show details about a plugin’s options)
10. -DskipTests this will skip running tests, but still compile it. or -Dmaven.test.skip=true which will skip compilation and execution of tests (use with judgment, I believe in testing)
11. If -X doesn’t show you the properties when you needed them, you can also use the ant-run plugin to echo a property value (does anyone else know a better way to do this in maven? Tell me what a property value is at the time of the plugin or goal executing?)
12. Run your tests paused in debug mode, waiting for a debugger to attach to port 5005 with mvn -Dmaven.surefire.debug test [4]

References:

• [1] http://maven.apache.org/plugins/maven-dependency-plugin/
• [2] http://maven.apache.org/plugins/maven-help-plugin/
• [3] http://cvs.peopleware.be/training/maven/maven2/debuggingPom.html
• [4] http://maven.apache.org/plugins/maven-surefire-plugin/examples/debugging.html
• http://puredanger.com/tech/2009/02/25/maven-tips/
• Personal experience

*Note: Initially I disliked the confusion and long, slow learning curve with maven. After working 8 months with maven and my colleagues Lucas Ward and Paul Hammant, I’m feeling much more comfortable and am generally okay with it. For small open source work, where you want a project quickly, but don’t want to spend a few hours with ant, I like it very much. For our large project, maven is fundamentally broken. Too much file copying is required, and manual intervention on a complex build is impossible, or extremely tedious. Running with our own build script we control could be dramatically more flexible.

On a mailing list Paul recently recommended a few other to consider:

Maven Dependency plugin
mvn dependency:tree
- shows me the transitive deps (say, “how the hell is MX4J being dragged into the build ?”)
mvn dependency:analyze
- show me unused, but declared dependencies.
There’s a ton of free capability to that plugin – http://maven.apache.org/plugins/maven-dependency-plugin/

An Atlassian dependency plugin
Atlassian wrote a plugin ( http://confluence.atlassian.com/display/DEV/FedEx+XI+-+Keeping+track+of+you+maven+dependencies ) that guards against undesired changing dependencies.

Maven Enforcer Plugin
A more intrusive way of guarding against deps that, say, should not make it into a war.  http://maven.apache.org/plugins/maven-enforcer-plugin/

What tips do you have? Please share them in the comments.

Reading time: 3 – 5 minutes

I’ve been heavily refactoring a large team’s codebase that stuffed many critical dependencies into request attributes. You see request.getAttribute() throughout the code, which is counter to dependency injection’s principles. Instead of getting what you want, be given it to you. We are moving to a more idiomatic way of using spring to inject things such as our custom Session object.

However, there was a lifecycle mismatch between some singleton interceptors which needed the session object, which is only request scoped. We had everything working, except one of those interceptors also needed the response object as a collaborator.

I had a spring instantiated non-controller bean that is request scoped, and I want to inject in two collaborators:

HttpServletRequest (which can be injected)
HttpServletResponse (which fails injection)

Here is my bean:
<bean id="sessionFactory" scope="request">
<aop:scoped-proxy/> <!-- scoped-proxy to get around the singleton requests injection of a request scoped object problem -->
</bean>


Here are the autowired fields:
@Autowired HttpServletRequest request;
@Autowired HttpServletResponse response;

Can I inject the response object in this bean?

This is the error I was getting:
[INFO] Caused by: org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name ’scopedTarget.sessionFactory’: Autowiring of fields failed; nested exception is org.springframework.beans.factory.BeanCreationExce ption: Could not autowire field: javax.servlet.http.HttpServletResponse com.example.web.SessionRequiring$SessionFactory.re sponse; nested exception is org.springframework.beans.factory.NoSuchBeanDefini tionException: No unique bean of type [javax.servlet.http.HttpServletResponse] is defined: Unsatisfied dependency of type [interface javax.servlet.http.HttpServletResponse]: expected at least 1 matching bean
[INFO] at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessAfterIn stantiation(AutowiredAnnotationBeanPostProcessor.j ava:243)

I navigated to GenericWebApplicationContext, which delegates to WebApplicationContextUtils.registerWebApplicationS copes(beanFactory), which then registers ServletRequest.class and HttpSession.class as injectable. But not the response object.

So here’s what I did, which works for me:

In web.xml add a filter that will store every ServletResponse in a threadlocal, which will be accessible only from your FactoryBean.

  <filter>
        <filter-name>responseInScopeFilter</filter-name>
        <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
        <init-param>
            <param-name>targetBeanName</param-name>
            <param-value>responseInScopeFilter</param-value>
        </init-param>
  </filter>
  <filter-mapping>
        <filter-name>responseInScopeFilter</filter-name>
        <url-pattern>/*</url-pattern>
  </filter-mapping>

Create the filter:

    /**
     * We need a filter to capture the HttpServletResponse object such that it can be injectable elsewhere in request scope.
     */
   public static class ResponseInScopeFilter implements Filter {
        // not the most elegant, but our spring commiter friends suggested this way.
        private ThreadLocal<HttpServletResponse> responses = new ThreadLocal<HttpServletResponse>();
        public void init(FilterConfig filterConfig) throws ServletException { }

        public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain chain) throws IOException, ServletException {
            HttpServletResponse response = (HttpServletResponse) servletResponse;
            responses.set(response);
            chain.doFilter(servletRequest, servletResponse);
            responses.remove();
        }

        /** Only to be used by the BeanFactory */
        private HttpServletResponse getHttpServletResponse() {
            return responses.get();
        }

        public void destroy() { }
    }

And also create the FactoryBean:

/** We need our own FactoryBean because Spring does not offer a way to
     * inject request scoped HttpServletResponse objects.
     * See http://forum.springsource.org/showthread.php?p=298179#post298179 */
    public static class HttpServletResponseFactoryBean implements FactoryBean {
        @Autowired ResponseInScopeFilter responseInScopeFilter;

        public Object getObject() throws Exception {
            return responseInScopeFilter.getHttpServletResponse();
        }

        public Class getObjectType() {
            return HttpServletResponse.class;
        }

        public boolean isSingleton() {
            return false;
        }
    }

Wire it all together in applicationContext.xml

<bean id="httpServletResponse" class="com.example.web.SessionRequiring$HttpServletResponseFactoryBean" scope="request"/>
<bean id="responseInScopeFilter" class="com.example.web.SessionRequiring$ResponseInScopeFilter"/>

Elsewhere I can now inject an HttpServletResponse object and it will be request scoped appropriately for this request. You can read more about this in my post on the spring forums.

Reading time: 2 – 3 minutes

At ThoughtWorks, we have many great technological projects, and brilliant coworkers. But beyond all that is a power for social good that is out at work in the world. Today, many of us are involved in ongoing projects with UNICEF and other non-profit organizations, helping solve fundamental human issues with technology. Last winter North America had an all-hands, and we brought in people like Merrick who’s helping the world with mobile SMS based apps at UNICEF. And we have Jeff Wishnie who is a Silicon Valley veteran, paragliding instructor, and now our Director of Social Engagement. He brings both socially aligned clients, and goes out and leads missions into all parts of the world using technology to better humanity.

ThoughtWorks has three distinct pillars that describe it:

• First it needs to be a sustainable business.
• Second it champions software excellence.
• Third it has a passion for social and economic justice.

Our founder and chairman, Roy Singam, sent a great email out a few days ago that inspired this post, and reminded me the importance of how each of us choose to spend our working hours. Over coffee and engaging life-beyond-mere-profitability conversations he gave me permission to quote him:

“Being part of an organization that advances a cause is an important. Associating with people who collectively encourage moral behavior is one of the most important decisions one makes in life. To ignore this and treat these decisions as simple career decisions (or give a little guilt money) is avoiding moral responsibility.”

We have ThoughtWorkers on the ground in real projects working with global and local NGO’s that allow us to directly apply technology, process design, and lean management to changing the world. The mobile space is especially exciting.

I believe future proof, profitable corporations, with reason beyond profit will retain the most capable employees, and provide lasting global impact as 100-year socially-positive companies. Profit, smarts, and growth is essential, but the meaning of work-life must extend beyond the bottom line.

Reading time: 2 – 4 minutes

Thanks to Roy’s urging, over the last 6 months I’ve been involved with London Business School’s Lynda Gratton in a consortium on the Future of Work. What will work look like in 2020, and how can companies become “future proof?” Specifically, what will global corporations look like? We kicked it off in London last November, where we met and discussed with people at several dozen companies. It’s been an honor, and a time to meet many interesting people.

Today, looking back at it, I am extraordinarily proud to call myself a ThoughtWorker. As an employer, we have a unique positioning of a global culture, lived out social values, and world changing technology.

Go visit any ThoughtWorks’ office in the world and you’ll find many people on international assignment. Further globalization and virtual-globalization will permeate work in 2020 — but today we clearly stand out as a leader amongst our peers. On vacation last winter in Beijing (pictures), I visited our office. It felt no more like I was in China then if I were in our offices in Bangalore, San Francisco, London or Chicago. It was not an “American culture,” but a “smart, interesting, passionate” culture of people with deep and varied interests — surrounded by great technology and delivering some game-changing products.

We encourage frequent cross-pollination of ideas and experiences through short and long term transfers. I felt right at home, met some amazing people (fellow TW’er and father of the Chinese internet Michael Robinson), got set up with daily Chinese lessons, and returned to visit in the office (for said lessons) almost every day.

There are so many more things to write about. It’s inevitable for work in 2020 to involve more cultures, countries, and languages. The marketplace for your goods could be half way around the world. Virtual-workers and virtual-meetings will explode. Yet, I also predict more face time with global coworkers. (Think how the world will change when we have US to Asia, or Europe to South America point to point travel in a few hours via spaceflight.) This will usher in increased complexity, and security measures; however I am excited for the future.

I have many more ideas, especially about the social values side of work in 2020 — but in true agile fashion, I’d like to see if anyone is interested in this before putting in more time up front.

Reading time: 2 – 4 minutes

McKinsey has an interview (pdf) with Kundapur Vaman Kamath, ICICI’s award winning MD and CEO from 1996 until 2009. He explains why he had no CIO, it was so strategic he brought it in as the CEO’s responsibilities. (An unusual move, considering the amount of effort two roles would stretch him very thin.) His boldness is evident in the following quote.

[Startups] in Silicon Valley were taking products from concept to market in 90 days, because if they didn’t, somebody else would. So we asked, “Why can’t we?” We made it a rule: no project was to extend beyond 90 days. People were skeptical at first, but it was achievable, and it gave us a huge competitive edge. When I first heard about the 90-day rule at a seminar, we were building a platform for online brokerage almost from scratch. I got on the phone to Bombay from New York and said, “We need to get this done in 90 days.” The project had already been going for 30 days, so in the end I said, “OK, you can have 90 days from today.” The trading platform was up and running 90 days later. It cost us just over $1 million, and with some marginal tweaking–nothing more–it is still operating today.

Imagine that! Every project must go to market in 90 days. What would your organization look like if you instituted such an aggressive policy?

If implemented in most organizations, I predict two outcomes:

1. Many projects would be canceled, saving millions of dollars.
2. Surviving projects would release incrementally and progressively. No big up front design, followed by years of waterfall. Instead iterative enhancements and frequent production deployments. You won’t build what you don’t need, and you’ll get customer feedback faster to deliver more of what the customers want.

His other quote was great as well:

We decided to run technology in a radically different way from anyone else, so we don’t have a technology department or a glorious title like chief information officer. There is no CIO. Technology is embedded in every business, and the head of the business runs the technology.

Closer business and technology interaction: a recipe for success.

ICICI is India’s largest private bank, who succeeds primarily because it can rapidly implement technologies giving it a competitive edge, says the bank’s chief executive, K. V. Kamath. Kamath, CEO of the Industrial Credit and Investment Corporation of India, considers information technology so central to the bank’s achievements that he manages it himself, without a CIO. Drawing inspiration from the culture and methodologies of Silicon Valley, Kamath has turned a stodgy industrial lender into a regional powerhouse with assets of $56 billion. Having learned to serve low-income consumers cost-effectively in India, ICICI now is exploring other markets.

Reading time: 5 – 8 minutes

Engineering something to be scalable, highly available, and easily manageable has been the focus of much of my time recently. Last time I talked about spiderweb architecture, because it has attributes of scalability and high availability, yet comes with a hidden cost. Complexity.

Here is a fictional set of questions, and my responses for the application architecture.

Q: Why does complexity matter?
JAW: Because when your system is complex, there is less certainty. Logical branches in the possible state of a system mean more work for engineers to create a mental model, and decide what action to take. Complexity means there are more points of unique failure.

Q: But my team is really, really smart; my engineers can handle clever and complex mental models!
JAW: That wasn’t a question, but I do have a response. Given a team at any moment in time, there is a finite amount of complexity that the team can deal with. Complexity can be in the application’s logic, dealing with delivering business value. Or, it can be in non functional requirements. If the NFR’s can be met with lower complexity, this will translate directly to more business value. A team will grow in their ability to manage complexity as they understand more and more of it, and team size can increase. Although those productivity increases can be used for business value, or complex architectures. And often, NFR’s can be met while still achieving simplicity.

Q: So how do I deal with a large, complex application which needs an emergency fix on one of the small components?
JAW: Yes, I know the scenario. You want to make a small change into production, but it sounds less risky to only push one part. Here’s my recipe for success: make every deployment identical, and automated. (Ideally push into production from continuous builds, with automated testing.) In the event of an emergency push into production, alter the code from your version control tag, and deploy that as you would every other push. My colleague Paul Hammant call non-standard, risky pushes “white knuckle three-in-the-morning deployments.”

Don’t make the e-fix a one-off, non-standard production push. Have the entire system simple, and repeatable. With repeatability and automated repetition comes security. Very flexible (read: complex), extensible (read: rarely tested day to day) hooks can be built into a system, in order to make it possible to push just one small component into production. However in reality unused code becomes stale, and when a production emergency happens, people will be so scared to try these hooks. Or if they do, there is a greater risk of a misconfiguration, and failure. Which will necessitate a fix of the failed fix which tried to fix the original tiny defect. More complexity. Blowing the original availability requirements out of the water.

Q: So, what is simplicity?
JAW: My definition says: Simplicity is the preference of fewer combinatorial states a system can be in. Choose defaults over

I recently read a quote from High Scalability, which I think gives a good definition of what simplicity is (emphasis added):

“Keep it simple! Simplicity allows you to rearchitect more quickly so you can respond to problems. It’s true that nobody really knows what simplicity is, but if you aren’t afraid to make changes then that’s a good sign simplicity is happening.“

[Caveat: some complexity makes sense, it's just too much in the wrong places increases risk. And there is a threshold everyone needs to find: how much risk, how much flexibility, and how much energy to devote to reducing the risk while keeping high flexibility.]

Update: Thanks to Lucas, for pointing me to an interesting article about second life scaling:

A preconditon of modern manufacturing, the concept of interchangeable parts that can help simplify the lower layers of an application stack, isn’t always embraced as a virtue. A common behavior of small teams on a tight budget is to tightly fit the building blocks of their system to the task at hand. It’s not uncommon to use different hardware configurations for the webservers, load balancers (more bandwidth), batch jobs (more memory), databases (more of everything), development machines (cheaper hardware), and so on. If more batch machines are suddenly needed, they’ll probably have to be purchased new, which takes time. Keeping lots of extra hardware on site for a large number of machine configurations becomes very expensive very quickly. This is fine for a small system with fixed needs, but the needs of a growing system will change unpredictably. When a system is changing, the more heavily interchangeable the parts are, the more quickly the team can respond to failures or new demands.

In the hardware example above, if the configurations had been standardized into two types (say Small and Large), then it would be possible to muster spare hardware and re-provision as demand evolved over time. This approach saves time and allows flexibility, and there are other advantages: standardized systems are easy to deploy in batches, because they do not need assigned roles ahead of time. They are easier to service and replace. Their strengths and weaknesses can be studied in detail.

All well and good for hardware, but in a hosted environment this sort of thing is abstracted away anyway, so it would seem to be a non-issue. Or is it? Again using the example above, replace “hardware” with “OS image” and many of the same issues arise: an environment where different components depend on different software stacks creates additional maintenance and deployment headaches and opportunities for error. The same could be said for programming languages, software libraries, network topologies, monitoring setups, and even access privileges.

The reason that interchangeable parts become a key scaling issue is that a complex, highly heterogeneous environment saps a team’s productivity (and/or a system’s reliability) to an ever-greater degree as the system grows. (Especially if the team is also growing, and new developers are introducing new favorite tools.) The problems start small, and grow quietly. Therefore, a great long-term investment is to take a step back and ask, “what parts can we standardize? Where are there differences between systems which we can eliminate? Are the specialized outliers truly justified?” A growth environment is a good opportunity to standardize on a few components for future expansion, and gradually deprecate the exceptions.

Reading time: 2 – 3 minutes

My glass screen broke by popping and spinning up outside of my jacket, landing glass-side-down on a bumpy pothole  a few weeks ago. It was right before going to China, so I didn’t have time to take it to an apple store. I covered the glass with a screen protector (to stop shards from falling off), and waited until I had more time. I even tried having a phone store in China look at repairing it, but my language barrier got in the way. They all kept trying to use styluses to touch the screen. Now that I’m back, I decided to repair it myself and here are my findings.

Order a replacement screen and touch sensor together. Only my glass was broken, but they’re replaced as one unit.

First watch these two helpful videos for instructions. Make sure you remove the glass top with a suction cup (thanks Chris!), don’t pry it. Also, when removing the (unbroken in my case) LCD, do not pry on it. Instead pry on the metal frame it is attached to.

Do not use too much heat when loosening the glue of the broken glass. This was my only mishap. I used a 2200 watt heat gun and warped and melted off a piece of the plastic frame. Then I spent an hour trying to reheat and bend it back. Also, watch out for repositioning the center button – mine went back in a millimeter lower on one side so it feels different. (Actually this is probably because of the warped frame.) Regarding the rubber gasket; be careful, but some damage may be unavoidable on it.

It took us about 2.5 hours to complete it, and I recovered from the heat gun mishap so that it’s not visible and everything fit back eventually. Plus it was fun to see the insides of the iPhone. Good luck!