Stories of my career
I started my career working as an Assistant Marketing Manager in Brisbane, but then a year later I took a big risk and moved back to Hong Kong to start my own startup with my all savings for the year. I see an opportunity to build a website which allows users to find restaurants nearby.
If I would do it differently, I would learn about the lean startup process. I did not know what is MVP (Most Valuable Product) back in the beginning, and I built a lot of cool features, such as real-time maps and geolocation for finding restaurants nearby. It ends up nobody using it at all as it was not an important feature. Most of the website traffic was driven from Google search, so I should have focused on SEO instead of geolocation search at that time. I would do differently by using the agile scrum methodology and iterate on the product, getting feedback from the customers and prioritising the features based on the feedback.
Later on, since my startup failed, I found a full-time job, working at an Australian consulting firm Industrie IT as a software engineer, my first client was Riot Games, League of Legends, which is owned by Tencent. The project was a redesign of the game store that buy skins for champions. I was the sole front-end developer responsible for the implementation and I was committed in the sprint to writing a feature, with CSS animation on the purchase button.
I underestimated the complexity of this feature, and I promised to finish this within one week. However, there were some risks, because the design of the animation and the assets are provided by the designer. We were working in different time zones, where I was based in Hong Kong and the designer was based in Latin America. There was little overlap time between our working hours and their progress was delayed since the product owner could not decide which animation works better.
One day before the sprint end demo session, I finally received the animation file, and it was provided using photoshop, which I did not have a licence to open. The format of the animation was not compatible with the legacy browser, which is the safari 4 in-app browser. The latest CSS syntax was not supported and it does not render correctly as expected. There was no way I was able to demo the finished animation, given that I was stuck in some technical issues. What I did was raise the blocker in the standup meeting for a status update to the team, so that it becomes visible that it was blocked. Also, during the sprint demo session, I was able to demo one of the buttons that implemented the animation, despite I wasn’t able to complete all the buttons. It is still valuable to get feedback from the stakeholders to see how one of the buttons would look and improve. And in the sprint retrospective, I self criticised myself for the underestimation of the effort and promised to improve in the upcoming sprint.
As a result, the stakeholders are able to provide valuable feedback on the progress, at the same time, understand the effort has been provided and attempted to deliver on time. Since the project was not urgent to release, the stakeholders are okay with the delay and appreciate my transparency of the status, instead of trying to hide the fault.
This project, it was using an old safari 4 in-app browser embedded in the adobe platform. My task is to implement the new design and launch it to China users. I made the mistake did not perform testing on all platforms, but only on Mac and Windows 7. During the beta launch, customers who are using Windows XP started to complain and said they got a blue screen of death from a computer crash.
What I did is to investigate the root cause. It was not easy, because the Windows XP crash blue screen did not give a lot of useful information on why the system failed. And it seems to be a deep platform issue on the operating system level, which I was not an expert on. I tried to ask questions on stack overflow, and comments are mocking me for supporting Windows XP, which should be deprecated. However, the majority of the internet cafes were still using Windows XP, so I had to support it.
Finally, I have to try to collect as much data as possible, including trying to set up a new virtual machine with windows XP to replicate the issue. More importantly, I was using a binary search method like in computer science, by commenting out half of the code, and seeing if it crash or not, to see which half of the code may be causing the problem. Then further comment out half of the portion and finally find the root cause was a library called font-awesome, for rendering icons. The issue could then be rectified using png images instead of icons. And I learnt the lesson to test my application on different platforms.
For the second client, I was helping my client to implement mobile applications, both iOS and android platforms. It was an IoT project with an app to connect with luggage locks. In the beginning, we started to implement both iOS and android apps at the same time. However, after the end of the sprint, the customer realises some of the differences in user experience due to the difference between the two platforms, such as the back button. Also, the startup wants to get investor funding so they want to get the app to be ready for demo as soon as possible.
Therefore, I have to make a short-term sacrifice and focus on developing one platform first on iOS instead of Android. This is because the main customers would be Apple users, as the product enters the high-end, luxury demographic. By sacrificing the android platform in the short term, I was able to deliver the iOS app at a much faster speed with more features, better proof of the concept and polish on the user experience. Once the investment funding was raised, I was able to spend more time on the android platform with double the speed, since it avoids some of the re-work and it has a more stable set of requirements within a shorter period of time. The investment fund raised benefits for the long-term goals and let the start-up sustain itself.
This client was a small startup on Bluetooth lock. My task was to deliver the mobile applications to connect to it. One time the client expressed that he is not fully satisfied with the sprint end release that I delivered. The expectation is pixel perfect, with the colour matching exactly like the design, gradients and animation of the buttons need to be perfect. I was not able to meet the expectations as I did not notice the details in the few pixels alignment.
I pay extra attention to the pixels and colour, using different tools to measure the hex values and making sure correct alignment on different phone screen sizes. I reserved time and asked the designer for help to notice any issues she spots. I sat down with our designer and the client and explained the reasons for the few deviations in the design from what the client directly requested.
Although it took a lot of extra time and effort for those minor changes, it does improve and meet the client’s expectations. The client thanked me for going the extra mile to make sure he gets a perfect outcome. The start-up with the size of 2 people. They are working on an IoT project and building luggage with a lock connected with Bluetooth to the mobile app. I was a software engineer helping the customer to build the frontend application.
After the first version of the app was built and ready, the hardware side of the project was delayed. Therefore, the product owner started to think about redesigning the app without even launching it to the market. He was not a designer but started requesting a change of the design, and a couple of days later, he would change his mind requesting a change on a different design, which could lead to inconsistency. I was frustrated by frequent changes in the requirements and re-work.
What I did is to coach the client with agile methodology. The gist of it was to launch something to the market and get feedback from the customer so that we can deliver and prioritise the main features. It is a common pitfall for new entrepreneurs to want to launch the perfect product in the market with the best user experience. At the same time, it became the biggest psychological barrier for the product owner to launch something on the market. Instead, I introduced an A/B testing tool, so that I could build two versions of the app, and collect user feedback, to see which one is better.
Therefore, I was able to get data on the design of the mobile app. I was not frustrated because the change has data to prove it is something better, rather than based on the customer’s indecisiveness. The customer is happy as well because he knows which version is better and does not worry about different designs and re-work, focusing on the feature that is most important to the customer. Being apologetic and actively sympathetic is important when handling such situations, and providing a solution as soon as possible.
Later on, I decided to move to a bigger consulting firm and change my job working in Accenture, the client was Cathay Pacific Airways. The project is a migration of the legacy hybrid app to the native mobile, with better design and performance. It was a very tight timeline because the old app on the Kony platform has a license expiring in a few months. I have to finish re-writing all the logic in the new platform.
Near the end of the old platform license expiry, we still got a lot of features unable to finish. It is a hard deadline even though we are running on an agile scrum methodology. In order to meet the tight timeline with the fixed scope, and not sacrifice the quality, I have to compromise and work overtime, weekends and public holidays. I planned a trip to go to Cambodia with my girlfriend, but I have to cancel the flight and hotel. As a result, I make sure one of my biggest customers meets its year-end goal. The costs it saves from the licence and the 5 stars review on the mobile app store made from that project made it all worth it.
The higher management put a lot of pressure on internal reporting. It was many hours lost of productivity and administration, instead of focusing on the delivery of values and features to the customer. I escalated the issue and asked the higher management to follow the agile scrum approach. They are welcome to join the standup and sprint demo introspective of the progress. It would be an anti-pattern to use the waterfall approach and reporting.
Eventually, the customer is happy with the progress. And the higher management appreciates customer satisfaction, thus reducing internal progress report meetings that waste time and productivity.
There was one important feature needed to build a customer to online purchase flight tickets from the mobile app. It needs to integrate with one of the booking engines, called Amadeus. The task I need to do is implement the online flight ticket purchase without any bugs. However, I made a significant professional failure by using an open source library to calculate the decimal places. Most countries follow ISO 4217 standards for currency precision, which can range from zero decimals to three decimals. For example, USD amounts have two digits to the right of the decimal, and JPY has none. I did not realise the problem when using the library and it went through the QA process without any problems.
When the mobile app was launched to the public customer, Cathay Pacific Airways start to receive complaints that the online purchase via the mobile app got charged ten times more than the original amount! Luckily it did not happen to all countries, only to a few one. Turns out, not all countries strictly follow the ISO standard, such as Russia Ruble. It was supposed to be 2 decimal places while the system returns and counts with 3.
I immediately inform my manager about the issue and estimated the impact. The impact would be customers paying 10 times the money to buy a flight ticket. I need to communicate to stakeholders, with the customer service department about the technical issues, such that they would be able to handle customer queries. I also need to think of a plan, to roll back the bug and patch the error. I would need to notify the customer and create a force update plan for the mobile app, apologies for the inconvenient costs.
As a result, the company needs to refund the customer and apologies. What I learn from the situation is to pay extra attention to decimal places, and the situation comes up in my professional career a couple of times working in the fintech industry. More importantly, I should write more test cases and think of all the edge cases, such as different currency decimal places to make sure the system is working correctly. It is really customer money and the bug could lead to significant financial loss if the edge cases are not being considered carefully. I was able to minimize the problem and resolve it within 1 week of time. There was some customer affected in production and requested a refund and compensation. What I learn is to be extra careful with decimal places as it is significant. And I take responsibility to resolve the problem.
In the next phase of my career, I switched job and working at EY, my client was HSBC at that time. It was difficult because it involved many different vendors and stakeholders. Every day, in the standup meeting, instead of timeboxing it to 15 minutes, it took more than one hour and moved to a discussion section without any agenda. And then someday, when there is no particular progress, the customer starts yelling and blaming the team member for not delivering. The customer turns the environment toxic instead of being productive.
What I did is to set up a 1:1 session, and coach the client on why timeboxing a standup to 15 minutes is necessary. It is not a discussion session for random topics and there is a business impact, given the consultants are billing the banks with a couple of thousand dollars per hour. I manage the customer by asking him questions and helping him to find a better solution.
His reaction was the first defensive. This was because he was quite senior with many years of experience in the industry before the agile practice was common in the industry. Therefore, I have to coach him and help him to figure out why he got this behaviour. Turn out the client was stressed about the delivery timeline and worried that the team could not deliver on time. The more he worries, the more time he spends chasing progress in standup and the less productivity. As I help him to find out the root cause, I help him by suggesting a better approach, which is to schedule discussion sessions offline if necessary and only include relevant members to join. Also, I remind everyone to physically stand up in the standup meeting, so that they would feel tired from standing and shorten the unnecessary status update.
The outcome of this was positive. The team appreciates the time saved to focus on writing software. The client was actually less stressed because the team is actually able to deliver more. I was able to transform the toxic work environment into a healthier environment so that the project was able to deliver on time eventually. The project manager ended up being great friends with me in the end, and appreciate my feedback to help the team.
After the app was developed and launched, the users were not happy and leave negative reviews on the app store. I have to investigate why the users were not happy with the app. Because the mobile app has defects, different from the original design and product specifications. Why? People the developer were outsourced to other countries and there is a loss in miscommunication. Why? Because the developer was not an English speaker and did not fully understand the requirement. Why not one find it out early? Because there was no significant QA process. So I ensure the quality by adding test cases and making sure it aligns with the original user story acceptance criteria. As a result, less bug and 5 stars review on the app as everything work smoothly with an improved user experience.
While working in EY, I also joined a hackathon and pitch an idea for a project to check suspicious bitcoin transactions. Users are anonymous on blockchain and use it for illegal activities, such as money laundering and drug dealing. I did a POC proof of consent project in an attempt to get the wallet address that has a high risk of suspicious activities. As a result, it got further implemented into real-world applications. It was innovative because it does help with the regulation of the technology and facilitates the development of fintech.
Moving on, I switched jobs due to a friend referral and I was working at Dynatrce, my difficult interaction with a client was the Hong Kong Jockey club. I was a consultant and helped them install a new software monitoring system in their data center. There was only one week available to complete all the installation and it was difficult because the client has a tight time window and only can do changes during summertime when there is no horse racing.
It was my first time working in a physical data centre, where it required a security check and it was freezing cold to work inside. There are multiple stacks of machines and I have to remember which one I was doing the installation and plug-in with the KVM (keyboard, video, mouse) system. After a week of successful installation of 16 different machines in 2 data centres, and before the horse racing day resumed, the security team performed hardening of the machine. And suddenly at midnight, I received an urgent call and request for production support.
I took a taxi to rush to the data centre and saw my client crying there. Turns out the security team removed the root users to access the database, and due to some misconfiguration, it did not provide the right user access to access the database for the software monitoring software. What I did in this situation is to calm down my client and start doing the re-installation. This is because we were running out of time and we had to re-install all 16 machines before the race day started. Therefore, I had to work overnight and at weekends, squeezing all the workloads from 1 week to 3 days.
Due to the hard work, the software monitoring software is able to complete the installation before the horse racing day starts. My client who was crying appreciated my effort to go the extra mile working on weekends and at midnight to help complete the task by re-doing all the installations.
My other big customer was Huwai. The project is to integrate the software monitoring product into their cloud environment. There are many feature requests coming in at the same time before the SoW is signed. I was trying to help the customer as much as possible and take the lead to aggregate all the requirements. They want to customise the frontend UI, however, our product is English only and did not have internationalisation.
I help to raise a feature request to the engineering team in Europe. It was a difficult process as they did not understand the necessity to customise the front-end UI. A lot of pushback happened and they said no capacity and it was not in priority. I escalated the issue as Huawei is one of our biggest customers in the China region. I got the attention of the CTO and got his support on this feature request.
As a result, the feature got delivered, with a customisable UI. When Huwai cloud users launch a virtual machine, there is an option for them to select if it comes with software monitoring, and there is a customised UI in the Chinese language. I listened with understanding and took them through the process step by step with patience and care. I focused on making sure that the customer clearly understands and was able to use the product in the Chinese language.
In this Huawai project, I need to integrate the software monitoring application into their cloud platform. I was a consultant sent to the Shenzhen onsite office, while my manager was in Australia, who did not understand the customer. The client’s request was to track the IP address of every user with personal identity information, which I have to say push back since this violates the GDPR (General Data Protection Regulation of the product.
I, personally, believe that there is always a solution to a problem. Regardless of how difficult it is, I will truly try my best to fulfil a request by a customer or at least meet them halfway. I would say no to a customer only if their request is simply not realistic, or expected me to directly break company policies. I asked the product engineering team to consider having a separate implementation, one for Europe customers who need to follow GDPR while one for China customers who have different regulations.
Consequently, the user frontend has to pass in a different header to identify as a China user. It would be difficult to analyse the IP by geolocation as the analysis may potentially violate the GDPR already. Meanwhile, it is also beneficial to the customer to think about the longer term and if they would consider a different implementation for global customers in different geolocation regions.
3 years ago, I leave my job and changed career to work at HSBC bank as a technical lead, the project I was working on was Malaysia FPX (Financial Process Exchange). It was a merchant online payment solution, where customers can go to Shoppee to pay and select debiting from their HSBC account using FPX. The situation was an existing platform using a legacy system, and I need to drive from improvement to increase the success rate.
The existing platform has a transaction successful rate of 30%. It was a very low rate, given that the regulatory requirement is a 70% success rate. My task is to increase the success rate by rebuilding the platform using a newer technology stack, such as Angular JS for the frontend and Java spring boot for backend applications.
After a few months of hard work in the project, the new platform was finally launched, I had a hope that the transaction success rate would significantly improve and reach the 70% target. However, it turned out the new successful rate was only 60%. I do not know why, so I have to conduct an investigation by using customer feedback. Firstly, I tried to use the tool called Splunk to collect all the transaction logs and count all the errors to identify what are the common error reasons, such as timeout. Also, I tried to reach out to the customer service team to find any user feedback. Turns out I found out we had an accessibility issue on the website. If you are a customer who is visually impaired, you would not be able to read the screen and know the transaction needs to be completed within 10 minutes. They take time and struggle to navigate the page, and it would time out and fail to complete the transaction.
Because of the feedback collected, I fixed the errors due to system issues and frontend accessibility issues. This increased the transaction success rate to 70%. This is important because it is a regulatory requirement from the Malaysian government, and it avoids a business impact on the bank by paying a large amount of penalty. The project also reminds me of customers who have accessibility issues, such as those who are visually impaired, any fix the details that would have every single customer. I have to approach every project with a clear and open mind to take the customer feedback.
During my time there, I was also proud to join an Open banking hackathon. I encourage HSBC to take the risk and adopt banking login as a service for other platforms. It could be security risky for the bank, but it meets the business goals since the bank account has already done all the KYC. I got to the senior CTO level and pitch the idea and show the demo. The CTO was impressed and my team win one of the awards for the hackathon.
When I was living in Hong Kong, I took a calculated risk and moved to Singapore. This is because Singapore has a lot more opportunities in financial technology in the Asia Pacific and it would fulfil my professional goal with overseas work experience. The trade-off requires me to move out of my comfort zone as I had a stable job in the bank HSBC, working as a technical lead in the corporate world. I have to leave my family and friends in Hong Kong to get to an unknown startup environment. It was a calculated risk because, in the worst-case scenario, I could still move back to Hong Kong if I did not enjoy the work environment in Singapore. However, the upside is unlimited, as I could see different opportunities and work environments, even in this challenging covid19 situation and required for 14 days of quarantine in the hotel. The outcome is I really enjoy the work environment in Singapore, it’s green, well planned and organised. There are also many work opportunities in the fintech industry.
Currently, I work at Thought Machine as a client engineering manager. It is a post-sales role, which means I am billable by the hours and service provided to the customer. Once the project is finished, and no longer billable, it is handed over to the customer and they are on their own.
However, in most of the scenarios, the client would have a need to ask some questions or production support queries. They would be able to contact me via the slack channel since I had built a personal relationship with the customer. At the same time, the needs of my business are to focus on billable hours and filling in timesheets for utilisation. The time it takes to answer customer queries would be out of scope and not billable.
To balance it, I would still try my best to find out the answer for the customer, even if it may mean I need to work some extra hours to compensate for it. It may not be billable right away, but the need of answering the customer’s questions should be a high priority.
Overall, client is happy to get the technical queries resolved within a short period of time. Instead of going through all the processes to raise support tickets, wait for a few days to get feedback. And after I have answered more than 10 technical queries, the customer was happy to compensate me by paying me for billable hours and satisfied the needs of both businesses. My approach is always thinking about long-term benefits for the business and retention of the customer rather than making a short-term profit and losing the customer. Sometimes, customers ask questions about a product issue. The balance was not deducted correctly. I felt they need an answer right now, but I think they need the correct answer instead of a quick answer.
I get on a call and set the expectation that I was trying to collect more information instead of resolving the issue right now, as I may need help from the UK team at different time zones. Communication is key here and I set the expectation that I will get back to them as soon as possible. And I have to work overtime staying late till the UK hours to help troubleshoot if required. Therefore, while the customer gathered more log information in the call, they realised something was wrong with the upstream system and sent the wrong balance amount in the request. The root cause was identified and resolved while trying to troubleshoot the issue.
One of my clients now is a Singapore Digital bank. After I delivered the python code for them to implement the current account business logic, it was tested and it was going to launch to production in beta. And before the launch, the monetary authority of Singapore would be required to perform load testing by a third-party auditor, which is EY. They were doing performance testing on AWS on our platform and when it hits 15 transactions per second (TPS) to one single account, they started to see response time degrade and get longer latency on response time. The customer raised a production incident ticket and I had to reply within the Service Level Agreement (SLA), as I was in the first line of support helping out the customer.
What I did is to request the relevant information quickly, including the Grafana dashboard for metrics data, kibana for logging data, as well as the latest source code from production. I was able to analyse the data and identify the main issue is a feature on user transaction daily limit. It was a feature where customers could only spend a fixed maximum amount per day, such as $1000 SGD. The key issue is the implementation was trying to loop through one day of transactions, and aggregate the total amount in order to check the current spending. It becomes an issue with 15 transactions per second, which becomes 900 per minute and 54000 per hour! I respond immediately to the situation by providing the root cause analysis and mitigation, where I changed the implementation by using a variable to store a record of the running total.
Therefore, the code was much more performant than the original implementation. I was able to resolve the production incident within a short period of time, and it helps the client perform the third-party audit check and complete the time-sensitive regulatory requirement within a short window.
Besides, the sales team will always ask me for an estimation on a prospect, and see how long would it takes, how many resources and how much it would cost for the prospect if they choose our project team to deliver the result. It usually required a quick judgement, since the prospect would only give a few days to respond and the requirement provided is incompleted and only a few lines, such as multiple currency support of the account with compliance, without any details.
I would need to make a quick judgement of the project estimation. Because if the estimation cost is too high, the prospect would not choose my company to deliver the project. However, if the estimation is too low, I would be in trouble as not having enough resources and time to deliver within the tight timeline. And usually, the prospect did not know what they want as well, and the procurement team is not the one who comes up with the product requirement.
What I did is try to approach the prospect, set up a call and clarify as much as possible. Then I would make some assumptions and validate them again with customer feedback, such as assuming the client environment has the infrastructure ready, the pipeline setup etc. And then I would also reach out to my other project team with similar project scope to collect data points on their delivery. Besides, I would make the best cases scenario and worst-case scenario estimation with some buffer to compensate for the lack of deep analysis of the requirement.
Eventually, the sales team was able to get back to the Vietnam prospect, and they were happy with my proposal. They selected my company product out of 4 other vendors for their core banking system. More importantly, the project was able to deliver on time because it was a realistic timeline and reasonable assumptions were made.
After I started working in my current job, I needed to learn about Kubernetes and Cloud technology since it is used in the product. I want to get certified and validated by knowledge, so I took multiple exams, such as CKA, CKAD and CKS for Kubernetes. Google Cloud professional architect certifications, Azure and also AWS cloud certificates. It took a lot of time to take online courses, watch videos, and reading blog posts, reading books about all the best practices. And more importantly, I do hands-on practice and try to solve real-world problems, applying knowledge in practical solutions. Recently, I got the certificates, not only learned the concepts but forgot about them after exams. Instead, I learn by doing and still use them to solve real-world problems.
My job now is client facing and need to do a product demo. After doing the product demo at a couple of sprint end, I set a goal for myself to improve my presentation skills and be a better storyteller. I recognised that communication was the heart of the job, so I really wanted to become a master communicator and impress my client. I spent a lot of time practising and joining the toastmaster club, paying attention to feedback on what did well and what did not do well, such that I can improve on it. I even enrolled on a storytelling class to help me advance. My efforts eventually led me to present a sales pitch in front of a Taiwan prospect, where I was able to show our product demo and answer technical queries. As a result, they selected my company’s product for their new project.
It has been a long journey since I started my career 10 years ago. I am writing all these stories as a practice to improve my communication skills, which is essential in order to reach the next stage of my career. Hope you enjoy the stories.
Originally published at https://www.victorleungtw.com.