Azure Data Factory (ADF) is great for extracting data from multiple sources, the most obvious of which may be Azure SQL. However, Azure SQL has a security option to deny public network access, which, if enabled, will prevent ADF from connecting without extra steps.
In this article, we’ll look at the steps required to set up a private endpoint and use it to connect to an Azure SQL database from Azure Data Factory.
‘Deny public network access’ setting in Azure SQL
Before we get started, let’s review which setting I’m referring to in Azure SQL. It’s a toggle named deny public network access found under Security > Firewalls and virtual networks in the Azure portal.
Source: author
When this setting is enabled, Azure Data Factory won’t connect without a private endpoint. You can see there’s even a link to create a private endpoint below the toggle control, but don’t use this now — we’ll create the request from Azure Data Factory in a minute.
ADF integration runtime
To use private endpoints in Azure Data Factory, you must use an integration runtime with virtual network configuration enabled. The setting cannot be changed, so you’ll need to create a new runtime if you don’t have one with it enabled already.
Source: authorSource: author
Now that you have an integration runtime with virtual network configuration enabled, you’re ready to create a new linked service.
ADF linked service
While still in Azure Data Factory, click to create a new linked service.
Source: author
When you select an integration runtime with virtual network configuration enabled, a managed private endpoint setting will appear in the account selection method section. The setting is read-only and will populate as you enter subscription and server details. If a managed private endpoint is already available — you’re good to go!
If a managed private endpoint isn’t available, click the create new link button to start the process.
Source: author
When you save the new managed private endpoint in Azure Data Factory, it will be provisioned in Azure but remain in a Pending status until approved.
Azure private endpoint
Now we need to hop back to Azure to approve the new private endpoint. Find your Azure SQL database in the Azure portal, and browse to Security > Private endpoint connections.
You should see the connection created by Azure Data Factory with the status Pending. Select its checkbox and click the Approve button.
Source: author
The status will change to Approved in the Azure portal. It takes a minute or two for the status to make its way to Azure Data Factory, but it will show as Approved there after a moment, too.
Source: author
Once it shows as approved, you’re ready to go. You can enter the rest of your connection info and connect!
Recap
Most of the settings I’ve shown can be accessed in several different ways and performed in different orders. For example, you could create the private endpoint from the Azure portal instead of through Azure Data Factory. You can obviously experiment and find the process that works for you.
The important pieces are the following:
Azure Data Factory has an integration runtime with virtual network configuration enabled.
Azure SQL has an approved private endpoint connection.
Azure Data Factory has a linked service using the integration runtime and private endpoint connection.
That’s it — now go have fun with your new connection!
This article was originally published in Towards Data Science on January 20, 2021.
Interested in learning more about Azure Data Factory? Give these books a try. Note that I use affiliate links when linking to products on Amazon.
There is no single right answer to any non-trivial problem in software engineering. So, if multiple correct solutions exist, how do you decide which is best? It’s difficult to determine which is best because “best” is highly subjective and deeply personal — your opinion is formed from your individual collection of experiences, strengths & weaknesses, and values on related aspects like simplicity, maintainability, & scalability.
It’s these internal values that make all of this so tricky. Imagine a spectrum with engineering excellence at one end and business needs at the other. Both elements are required for a project to be successful, and operating at either extreme can be detrimental to the other. As an example, making too many quick-twitch fixes to address urgent business needs can have significant long-term impact on the quality of the code base or system maintainability; conversely, focusing too deeply on engineering excellence can lead to over-investment in areas or competitive disadvantages from being slow to market.
Understanding this spectrum — and having awareness of where you and your colleagues lie on it — can help your team to be more pragmatic.
Awareness of this spectrum alone isn’t going to do you any favors in resolving conflict from perceived disconnects between you and co-workers, though. I’ve found myself in design/requirements stalemates many times, and I’ve used the spectrum as a way to visualize my frustration.
“You see, I live over here on one end of the spectrum,” I’d say, “and my colleague operates here, at the other end. We can’t agree on scope, and we aren’t getting started or making any progress as a result.”
The problem with the visualization as a tool for conflict resolution is those pesky personal values. Neither of us thinks we’re advocating for a solution that would be in the unhealthy extremes of the spectrum. The person in the engineering excellence camp just believes that business value is generated by following all the best engineering principles and creating scalable, high-performing, resilient applications whereas business needs nation wants quick delivery and maximum responsiveness to meet the ever-changing needs of its customers.
So, how do you find compromise when the source of conflict is so visceral?
Let’s see if we can steal a page from the Goldilocks playbook. She’s got a knack for identifying the undesirable ends of a spectrum before settling into a satisfying sweet spot. If you and your team or colleague(s) can’t agree on the scope of a solution, can you agree on what it shouldn’t be?
What’s a reasonable solution that everybody agrees is over-engineered, and what’s the fastest, but perhaps short-sighted, thing you could do? What’s the effort required for each approach, and what are the risks or consequences?
Just to be clear, I’m not suggesting to simply compare different proposals by plotting them on the spectrum— that probably won’t get you anywhere. Instead, work collaboratively to come up with bad solutions that lean too far in both directions. Find agreement by identifying undesirable characteristics of these options in the unhealthy parts of the spectrum.
Still not able to find compromise? It’s probably time to bring in a 3rd party, preferably a stakeholder. Show them your spectrum and explain the tradeoffs that exist at the opposite ends, then present the “real” options that are on the table and allow the stakeholder to decide.
The whole activity is an exercise in pragmatism. How can two parties with equal but conflicting opinions find common ground? The key is to calibrate and remove as much subjectivity as you can. By acknowledging the necessity of both aspects — engineering excellence and needs of the business — and agreeing on the “bounds” of the spectrum, you create a framework for identifying the region for compromise. That’s your sweet spot. That’s likely where your “best” solution should be.
Originally published at The Startup on September 21, 2020.
I grew up as a computer kid in the 80s and 90s and, consequently, spent a lot of time in video games. Now I manage two software engineering teams, and it’s time to prove to Mom that all those hours spent in Final Fantasy were actually valuable career development.
I’m primarily thinking about two video game genres: party-based RPGs and MMORPGs. The formula for these is pretty simple. You have a cast of characters with various capabilities, and they work together to accomplish amazing things. In order to succeed, you must be aware of your characters’ strengths & weaknesses, understand what skills are required to complete a challenge, and combine characters in a way that allows them to achieve the goal.
Well, that doesn’t sound so different from a software development team, does it? You’ve got a group of people with different abilities and aptitudes; you have a backlog of stories to complete; and the team must collaborate to achieve goals and accomplish amazing things. It’s, like, the same thing!
Given these undeniable parallels, what lessons from RPGs can be applied to software development teams?
It’s Dangerous To Go Alone
When you adventure alone in a video game, bad things don’t happen most of the time — but there’s risk. The same is true in software development, especially if individuals on your team are all working on separate things. The primary risks of soloing a software development project are consistency, quality, and knowledge-sharing.
Working alone on projects offers short-term risk due to the fact that quality and completion time are largely dependent on who does the work. Someone who’s very experienced and understands the subject area well will probably get by fine, but a developer with less familiarity will take longer and be more likely to make mistakes that require re-work. Peer code reviews are slower and less effective because reviewers need to context-switch and focus deeply to understand decision-making and what’s been done. Individual stories are completed less efficiently because everyone has their own assignments and agenda, and other people’s work becomes a secondary objective.
Knowledge-sharing and information silos are the long-term threat. It’s easy for people to become specialized as certain types of work gravitate toward them. If an individual is the only one that worked on a specific project, guess who gets tapped when it needs attention later? Future work can also be bottlenecked when the person with all the knowledge isn’t available, and the situation gets worse when that same person is the bottleneck for multiple workstreams or when personnel changes occur.
In the video game world, the risks and consequences of soloing are typically limited to just you. People choose to solo because it’s convenient — they don’t want to wait or look for other players — or because they enjoy the challenge. Sometimes it feels easier to go it alone in software, too. However, software development is a team game, and you’re not typically looking for extra challenge just for the fun of it.
Ensuring you have multiple people working on a project mitigates the risk. One person doesn’t go down a bad path by themselves, and at least two people should know how things work and why decisions were made. Pair or group programming is inherently review-as-you-go which leads to better initial code quality, and that in-turn helps with completing stories efficiently due to less feedback cycles and re-work.
You don’t want everybody acting alone, but it’s equally important to make sure you don’t have too many resources focusing on a task. In other words…
Quests in RPGs require a certain set of skills to complete. Easy quests are achievable by smaller, less experienced groups, but hard ones require more people, specific skills, or other special assistance. In games, it may seem more efficient to clear easier, low-level content with powerful, advanced characters, but if the strongest characters are focused on easy things, it means they aren’t working on more challenging, higher-reward objectives.
Software development is an exercise in efficiency. You have a backlog filled with user stories. Some are easy and some are difficult. You want to complete them all, though, and the faster you can do it, the better. The trick is for the team to determine the optimal way to complete as much as it can as possible as quickly as possible.
How would you do that in a game? First, you’d decide which quests are most important and what skills they require. Next, you’d look at which characters are available and what skills they possess. Then you can optimize who can do what. Perhaps one small group could tackle three easy quests while another group works on a single complicated one.
This same strategy can be applied to sprint planning. You’ve got stories in a prioritized backlog and a team of people to complete them. Which people have specific skills or knowledge required to complete the highest-value stories? Assign those folks first. Who’s left, and what’s the best way to utilize them? Make sure everybody is assigned, and balance the groups.
You may find that you don’t have enough people with the right skills to succeed with all the most important things. Luckily, RPGs give us a solution for that, too!
Characters in RPGs progress by completing tasks that reward experience. Once you’ve accumulated enough experience, you level-up and become stronger or gain powerful new abilities. In order to have a well-balanced team, you must use all your characters so they all gain experience. If you have a weaker, low-level character, you can grow them most quickly by leaning on them as heavily as possible in content that’s within their reach. Sometimes that can be painful, if you need to return to a low-level area and perform low-value activities, but the investment pays off with time. Once those characters “catch up” they provide valuable versatility to the team.
On a software team, this means ensuring that everybody’s getting reps with the most important skills. People won’t suddenly gain project management skills by not managing projects. Instead, acknowledge it: “I want you to manage this project so you can develop these skills.” You want someone to be a better decision-maker? Ask them to make decisions. Similarly, just because someone can do a thing in 15 minutes doesn’t mean they should do it simply because it would take someone else 2 hours. Instead, invest that 2 hours. The experience is as valuable as the time saved, and it contributes to both the growth of the individual and the strength of the team.
To be a successful team, there are multiple roles that need to be filled. As employees gain experience, they typically focus on a single role. For example, “I’m a developer, and this is what I do as a developer.” Often times, that’s enough to do a good job, but a great team member understands all the roles within the team and has the ability to recognize and step in when a role isn’t being fulfilled.
How do you develop that level of situational awareness?
Often times, games have encounters with mechanics that need to executed in order to succeed. You can win by knowing enough to go through the motions, but the best players don’t just know what to do — the know why they’re doing it. That also means understanding the consequence for not doing a particular mechanic and being able to adjust on the fly when things start to go sideways.
Software teams have a process designed to help the team succeed or — more pessimistically — prevent it from failing. Individual contributors can be successful by adhering to the rules and following the process. Great teammates will understand the underlying reasons for the process, though, and be able to make decisions around when it’s time to deviate.
Much like in games, a great way to become more familiar with the intricacies of your process is to learn it one role at a time from people who are already proficient. How does your team gather requirements and translate them into actionable work? What’s the best way to do a peer code review? What tests need to be run to verify there are no regressions? With better understanding, you can help improve these processes. Don’t just learn how to do things; learn why to do them.
These lessons from RPGs aren’t necessarily special or specific to software development; they’re just general best practices for teamwork and growth.
Recognize the strengths and weaknesses of individuals on the team, and utilize them in a way that makes sense — just like you wouldn’t have your squishy wizards standing in front of heavily-armored knights in a game.
Use the right number of people based on the task. There are both short and long term risks associated with using too few people but possibility for reduced efficiency & throughput with too many. It’s also important that the group have the right set of skills to accomplish the goal. Don’t poke the dragon alone, don’t bring the entire village to feed the horses, and don’t send a small group of adventurers into the dark cave without a torch.
Build your team by ensuring that people have the opportunity to grow. Acknowledge that having somebody less skilled perform a task pays dividends as they gain competency. If you don’t give people the chance to improve, they won’t. Having more people at level cap lets you tackle a wider array of challenges or accomplish more things at once.
Invest in helping team members understand the big pictures. What’s the greater purpose behind your team? Why do your processes exist? Encourage people to learn new roles and step outside the bounds of their specific job. The increased awareness and versatility differentiates good teammates from great ones. In an epic RPG boss fight, you can still win if somebody knows how to kite when the tank dies.
Know the team, grow the team. Use the right skills to get the kills, and collect that sweet, sweet loot.
It’s not uncommon for successful software developers to find themselves in leadership positions. There are many possible leadership trajectories, one of which is management. Moving into management was scary for me, and over the years I’ve talked to a number of people at that point in their careers experiencing a similar dilemma. This is the story of my experience, why I made the decision I did, what I’ve learned along the way, and how it’s turned out.
The pre-management era
Let’s start with a little about my background, eh? I started my career as a junior developer and worked my way into a senior role, eventually becoming an architect. As an architect, I served as the technical lead for my team and worked closely with managers to make various leadership decisions. I wore many hats in order to best address whatever I felt my team needed most at the time.
After nearly 10 years with my first company, I left to join a former co-worker at a startup. At the time, this new company was small enough that we didn’t have much of an organizational hierarchy. We were a group of senior developers that all had decision-making authority with freedom to work on what we felt was most important. We still collaborated, of course, but we weren’t all chipping away at a common product backlog.
The loose structure worked great for us as a small team, but it doesn’t scale. As we added more people to the team and new products to our catalog, we needed more structure. We divided into product-specific teams with narrower focus and dedicated backlogs. That was our no/low-management tipping point, and that’s where my journey into management begins.
My boss was suffering the consequences of our very flat organization, and they needed people willing to take on some of these managerial responsibilities. I felt like I was being forced to commit: did I want to be a manager or not? I wasn’t given an ultimatum; in fact, my boss was very clear that there was no wrong choice, and my career would continue to grow regardless. That was comforting, but I still had the decision to make.
The management path was scary for a variety of reasons. I had a proven track record as a developer. I was good at coding and troubleshooting. I knew how to solve software problems. Managers take on an entirely different set of problems that require a different skills — skills I wasn’t sure I’d have. Managing a team when things are going well didn’t seem so bad, but it’s the whole “dealing with people” thing that had me worried. Did I really want to give up coding — the thing I enjoyed and had built a successful career doing — to be a manager and have to deal with people?
Making the decision
I ended up lingering at this juncture for a while. Depending on the day, I might’ve leaned one way or the other, but for the most part I remained undecided and non-committal. In my free moments, I’d research what made a good manager and read stories like this one to understand other people’s experiences. There were a few nuggets of wisdom that helped me make my choice.
Skills can be learned. It feels obvious to say, but hearing this was reassuring because I always considered “learning new skills” to be one of my strengths. It gave me confidence to know that I could supplement applicable existing skills with new, learned skills. For example, I know how to troubleshoot an application by stepping through code until I find a problem, then come up with a solution and implement it to fix the issue. The same critical thinking can be used to identify problems with a team, but I may need to do research on agile processes or collaboration techniques to come up with the solution.
You can nerd out on management. As a developer, there are always new things to learn and play with: languages and language features, tools, technologies, and techniques. I remember learning about something like Microsoft Azure for the first time and being so excited to have a reason to use it. The same kinds of continuous innovation exist for managers, too, but you need some awareness of manager & team problems in order for the solutions to make sense — just like you need context for those new things in software development. How can you automate team processes to reduce toil? What things cause the most friction or prevent work from getting done? How can you make 1:1s more effective? How can you maximize the team’s impact on company goals? How can you improve individual accountability? These are all super valuable areas to focus on, and they’re ripe for fresh ideas and innovation.
There is no “point of no return.” I was worried I’d go from awesome dev to mediocre manager, be unhappy, lose my edge, and feel trapped. It was reassuring to hear that I could go back and that my development skills wouldn’t disappear after being less active for a bit. I’m less involved in daily development tasks now and there are a lot of pull requests doing thing that I don’t understand, but when I do get back into the code, it all comes back.
As you may have guessed, I decided that moving into a manager role was the right move for me, and I’ve been at it for a few years now. It hasn’t always been fun or easy, but the more I do it, the more proficient I become. This has a cyclical effect, too, because increased proficiency leads to more engagement which brings more enjoyment and satisfaction.
Learning to manage
The most difficult thing about becoming a manager for me was the self doubt. (See imposter syndrome.) I felt like I was less valuable to the team “managing” than if I was a developer focused on getting things done. I wasn’t sure if I was spending my time on the right things, and the things I was spending my time on were different than that of my peers/other managers. I was failing to adequately help team members that weren’t meeting expectations. I was doing the best I could, and my boss was supporting and encouraging me — but I didn’t feel like I was doing a good job. It was tough.
Looking back, I think there were two key mistakes I made as a new manager. The first was not being direct with people. I was trying to focus on positive behaviors and encouragement and not being explicit about areas that needed improvement. The second mistake was that I didn’t think about the strengths and weaknesses of individuals on the team and strategize how to use them most effectively. (More on that, here!) You can imagine how these two mistakes can create problems: praise for strengths versus less feedback & higher expectations for weaknesses. It’s no surprise that situations didn’t improve!
The single most important advice I have for new managers is to lead with empathy. Get to know your team, and treat them like people. Give them positive and negative feedback. You’d want your boss to tell you if did something wrong, right? But, it would start to feel bad if they only talked to you about the things you did poorly, too. Learn details about their lives outside of work. It would get annoying if you had to re-explain every week why you had to be late to the same meeting for same reason. The better you can understand your team — professionally and personally — the more effectively you can manage.
The best way to get to know your team is through one-on-ones. One-on-ones are a pretty common “most important thing” for new managers, but the thing it took me a while to learn was how to prepare for one-on-ones. This is very hand-in-hand with leading with empathy. Being prepared for these intimate meetings is the best way to demonstrate that you care. Come up with questions and topics that are specific to the person you’re talking to. What do they like or not like about their job? What are their goals, and are there things you can do to help ensure they’re progressing toward them? What are things they did well or poorly? What do they think they did well or poorly? It’s also a chance to solicit feedback from them about you. Be sure to take notes! They’ll help ensure you don’t miss important follow-ups and make preparing for the next meeting easier.
The next steps
Managing with empathy and preparing for & conducting one-on-ones are things you can implement immediately regardless of experience. The next steps take some time and depend on your team. The most important things will be to establish behaviors that will maximize effectiveness and define a clear vision and purpose. These are no easy feats, though. They require thoughtfulness, focus, strategy, and persistence & dedication. You must monitor progress and be wary of regression, and advocate for the right amount of change to keep things trending positively. And, you need to do all this in addition to whatever your team needs to keep things going operationally!
This is a good summary of what my day-to-day looks like now. I’ve got meetings and operational concerns, and then extra time goes into reflection and solutioning for these bigger, longer-term items. There are still ups & downs; still days where I feel like I’m doing a bad job; and still days where I wish I could just write some code. It’s been a rewarding experience, though. It’s been amazing to watch the team adapt & succeed and to help people grow. I love when we can take on an ambitious goal and achieve it.
Making the transition to manager was a scary decision. I had to learn new skills to solve new types of problems, and I had to battle through some self-doubt. I don’t regret my choice, though. I’m not perfect, I make mistakes & bad decisions, and I’m still learning as I go. But, I love my teams, and I love the problem space that I get to live in.
Have you made this transition yourself? What things have helped you succeed? Are you faced with the decision now — what worries you? I’d love to hear from you!
Originally published at The Innovation on November 7, 2020.
There are millions of articles and books about the importance of one-on-ones for managers and leaders. When I was a new manager, it’s something I read about and put a lot of effort into. I didn’t feel like I was good at them, and I didn’t look forward to them.
But — like many things — the more you do them, the better you get. This is the story of things I’ve learned with time and experience.
Make Time to Prepare
This is the piece of the puzzle that I was missing for too long. The secret to good one-on-ones is preparation. Make it part of your morning routine to prepare for each one-on-on you have that day. Eliminate distractions and dedicate focus to each person you’re meeting with.
The secret to good one-on-ones is preparation.
How do you prepare for these meetings, though? It’s a combination of looking back and looking forward. Review notes from the previous meeting, reflect on interactions and contributions, and think about the person’s life outside of work.
First, you need to make sure you’ve dealt with anything you signed-up for last time to demonstrate accountability and build trust. If you missed something, that’s okay — there’s still value in bringing it up as something you couldn’t get to, provided that you don’t lose track of things and deliver more times than you don’t.
Next, think about the person and how they’ve been since you last met. Have they completed important work? Has there been conflict? Have they raised complaints? Have they been engaged? Have they been visible? As part of your preparation, identify positive and negative things to call out. These can be individual or team items, too.
Finally, look ahead to the future. What short and long-term goals does the person have, and are there things you can do to help them make progress? Some examples would be to follow-up on a commitment they made last time, give feedback on a instance where they exhibited a specific positive or negative behavior, or provide advice or reference materials that could help them.
The one-on-one is also your time to solicit feedback from them. How do they feel about recent changes the team has made? How is the current project going? What do they think about the direction of the team? You can — and should — also ask them about you. What could you do to help them or the team? What would make you more effective as a leader or manager?
Preparation checklist:
Did you take care of action items from last time?
Are there positive or negative things to call out?
How can you help with their goals?
What feedback do you want to solicit?
Be Persistent and Dedicated
Preparation is critical for effectiveness, but the most important thing is to actually have the meeting. Sometimes the biggest hurdle is just getting the meeting on your calendar. I’m an introvert, and one-on-ones are mentally and physically exhausting. The thought of another recurring meeting on my already-busy calendar was demoralizing. Don’t procrastinate. You know you need to do it, so just get it on there!
Making the appointment is nothing without keeping it. It’s important to put forth your best effort to make sure they happen. Don’t treat them as your lowest priority. Things will inevitably come up, and you’ll need to reschedule your one-on-ones sometimes. When that happens, favor rescheduling over canceling, and reschedule sooner than later; not showing or canceling at the last second sends the message that you don’t care.
It’s not always easy, but force yourself to have the one-on-ones. Arguably, the harder it is to have the meeting, the more important it is. Relationships that are most strained are ones that stand to benefit most.
Take Notes
If you’re anything like me, you’ve got a lot of meetings with a lot of people. It’s impossible to remember all the commitments and action items in my head. Take notes! Taking notes has multiple benefits beyond simply having something to refer back to, more so if you write by hand. It forces you to listen and be engaged which helps demonstrate to the other person that you care and makes distractions less… distracting. Research has shown that writing things down helps you remember, so you’ll naturally retain more of the conversation. And, lastly, taking notes gives you the obvious benefit of having something to refer back to. Taking notes is immensely valuable even if you never look back at your notes.
If you want to be a next-level note-taker, take handwritten notes during your meetings, then immediately re-organize them into a digital format that you can refer back to during future preparation and that can also represent a running history. Make sure to flag action items or mark things for follow-up, too. Depending on how you work and organize, you may want to add items to your calendar or task lists. The “secret sauce” here is that you’ve now written it twice — so you’re less likely to forget it, anyway — and you’ve got some fail-safes in place to make double-extra sure that nothing slips through the cracks.
Set the Agenda
I’ve covered preparation, scheduling & commitment, and note-taking during the meeting, but what about the conversation itself? Here are some things to keep in mind as you conduct your one-on-ones.
Let them go first. This is the standard one-on-one format: they get 15 minutes then you get 15 minutes. Their part is your chance to listen, take notes, and understand what’s happening with them as a person. If people aren’t opening up, try asking questions like, “How are things outside of work?” or give them a choice of where to focus like, “How’s life, at work or away from work?”
Prioritize topics. Treat them like people first. If last time you spoke to them, you learned they had sick family, don’t jump right into grilling them about the reports they’ve been working on.
Be direct. If somebody isn’t meeting your expectations, say, “You aren’t meeting expectations,” and explain your expectations and why they aren’t being met. If you don’t address problems, they won’t improve. If you only focus on positive things to encourage good behaviors, it’s going to be received as, “[you] didn’t have anything bad to say!”
Highlight positives and negatives. It’s easy to worry about negative feedback and improving performance so much that you forget to celebrate the good things. It can be a drag to only hear about what you do poorly, so be sure to shine a lot on positive behavior, personal growth, and team victories.
Whether you’re a new leader or you’ve been at it for years, one-on-ones are an effective way to build trust, gain influence, and promote team & individual growth. It doesn’t come free, though. You need to do your part to ensure the effectiveness of these meetings, and that means more than just showing up.
Prepare. Commit. Take notes. Set the agenda with empathy. Do these things, and you’re sure to maximize the value of your one-on-ones!
I’d love to hear from you! What tips am I missing that make your one-on-ones most effective? What differentiates an okay one-on-one from a great one, or what things have you seen from the best leaders in your career?
In the first article in the series, we installed Python & Selenium and used them to write a script that opens a browser. In today’s article, we’ll look at how to do some basic inspection of web pages and modify our script to interact with them.
Selenium can do most of the same things you can do with web pages–namely, clicking & typing. But, just like you read and interpret what you see to know what to do, you need to tell your script how to find and do the things you want it to.
The easiest way to do this is to use the “inspect” functionality that exists in most modern browsers to help find identifiers you can use in your code. Let’s try it out with everybody’s ol’ buddy Google. Browse to google.com, right-click the search box, and pick Inspect.
This will take you to the selected element in the web page’s code. This is the part where you start to get some choices. Selenium has a lot of different ways to find elements, so you need to know about what you can use and compare it to what’s available. Usually the most reliable way to get an element is by using its id.
Usually, using an element’s id property along with Selenium’s find_element_by_id method is the most reliable way to find the element, but we don’t see an id when we look at Google’s search box.
We can see a name property with the value q, though, and there’s a find_element_by_name in Selenium, too; so let’s use that! With the element selected, you can use Selenium’s send_keys method to simulate typing. Add the following lines to your script:
When you run this script, the browser will open and send the specified text to the search box. Now we need to submit the search. Google has a search button, so we can repeat the steps to find the button and click it:
But, that’s not really how you search with Google, right? No–you type your search and press enter. To simulate this, you can use the submit function on the search element:
Run the script again, and you’ll see your search submitted and search results displayed. Cool stuff! Let’s do a quick review of all the tools we have now:
Browse to sites using webdriver.get
Find elements using webdriver.find_element methods
Enter text with webdriver.send_keys
Click things with webdriver.click
Submit forms–like you would do by pressing enter on your keyboard after entering data–with webdriver.submit
Have you ever been doing something on the internet and been like, “Ugh. I wish there was a way to do this automatically.” Well, most of the time there is: you can build some website automation using Python and Selenium. In part 1 of this series, I’ll walk you through installing Python and Selenium, and we’ll write a short script that opens a browser.
Install Python
For folks that don’t know, “Python is a scripting language that lets you work quickly and integrate systems more effectively.” (python.org) That means you write code in files and run the files without having to compile and deploy. In order to run the files, though, you need to install Python on your computer.
One way to do this is to download the installer from the official Python website, here. However, I prefer to use an application called PyCharm. PyCharm is an IDE (integrated development environment) available for Windows, Linux, and Mac that gives you lots of nice helpers when writing scripts, and there’s a very nice free community edition that’s just perfect for people like you who are just getting started.
Once the install is complete, you can launch PyCharm. The first time you run it, PyCharm will ask you about which theme you prefer. Pick what you like and click Skip Remaining and Set Defaults.
Create Project
PyCharm is ready to go, but we need to do a little more setup before we’re ready to write our code. Begin by creating a new project from the PyCharm launch screen.
Give your project a name to specify where it will live on your computer, and click Create. If you don’t already have Python installed (the actual programming language) at this point, PyCharm will install it for you.
Install Selenium + Chromedriver
Selenium WebDriver is what we’ll use to launch and control the browser in our script. In order to use it, we need to install the libraries. We can install it directly from PyCharm by running a command in the terminal.
Open the terminal by clicking the button.
Now run the following command:
pip install selenium
Selenium also needs an extra application to help control the browser. In this example, I want to use Chrome as my browser, so I need to download ChromeDriver from here. When you visit the ChromeDriver page, it offers a few different versions. You need to pick the version based on the version of Chrome you have installed. This can be found in Chrome’s About page.
Once you determine which version you need, download the Zip file for your OS and extract chromedriver.exe to your project directory. You can verify that it’s in the right spot because it’ll be visible in PyCharm.
Write Script
Flip up those hoodies because it’s time to code. Add a new Python file to your project by right-clicking the project folder.
Add the following code to your script:
from selenium import webdriver
driver = webdriver.Chrome('chromedriver.exe')
driver.get('https://www.google.com')
The first line tells our script that webdriver comes from the Selenium library that we installed. Then, it creates a variable named driver (but we could name it anything) that’s a Selenium WebDriver object using the ChromeDriver executable we downloaded. The third line uses driver to browse to www.google.com. Nothing too crazy, right?
Run Script
Go back to your terminal and run your script like this:
python <your-script>.py
When you press enter, a Chrome browser should open and navigate to the URL specified in your script. Chrome also gives an indicator that the browser is being controlled by automation software.
Pretty cool, right? There’s not much to experiment with this script. You can change the URL or add more driver.get lines to visit multiple sites. In the next article in the series, we’ll look at how to use WebDriver to interact with web sites and do things.
I recently started working on a new personal project that needed a database, and I just wanted something quick & easy. I remembered a colleague mentioning Firebase in the past and decided to take a look. After playing with it for a few days, I really dig it. I love that you can set up a database for free within a few minutes, and access it directly from the frontend without having to introduce a separate API layer.
In this article, I’ll walk through steps required to create a new Angular application and connect it to a Firebase Cloudstore database. You’ll need to sign up with Firebase to create the database, but it’s free and painless. Let’s get to it!
We’ll begin by creating a new application using the Angular CLI:
$ ng new firebase-app
$ cd firebase-app
Next, we’ll create our Firebase Cloudstore database. Go to firebase.com, and create an account or sign in. Add a new project and Cloudstore database in test mode. Finally, add a web app to the project. When you do this, you’ll be presented with instructions for adding the Firebase SDK to your app, as shown below.
Copy/paste the configuration details into our applicaton’s app.module.ts file. (Don’t worry about putting it in the right spot–we’ll do that in a minute. Just get it in there so you have it.) Now it’s time to install the Firebase and the Angular Firebase module:
$ npm -i firebase @angular/fire
When you install @angular/fire a browser will open for you to authenticate with Firebase, and you’ll be given an authorization code. Copy/paste the code back to the command prompt to finish installing the module.
With the module installed, we complete the necessary changes in app.module.ts:
import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { AngularFireModule } from '@angular/fire';
import { AngularFirestoreModule } from '@angular/fire/firestore'
import { AppComponent } from './app.component';
import { DemoComponent } from './demo/demo.component';
// Your web app's Firebase configuration
var firebaseConfig = {
apiKey: "AIzaSyAx2cAfq9Pj3EzavXLkNc6_F9zWCyIayY4",
authDomain: "ap-sample-1218e.firebaseapp.com",
databaseURL: "https://ap-sample-1218e.firebaseio.com",
projectId: "ap-sample-1218e",
storageBucket: "ap-sample-1218e.appspot.com",
messagingSenderId: "200601572991",
appId: "1:200601572991:web:a335d1e106542870a9914a"
};
@NgModule({
declarations: [
AppComponent,
DemoComponent
],
imports: [
BrowserModule,
// Initialize Firebase
AngularFireModule.initializeApp(firebaseConfig),
AngularFirestoreModule,
],
providers: [],
bootstrap: [AppComponent]
})
export class AppModule { }
At this point, we’re actually done with setup and fully functional, but let’s add a new component to demonstrate its usage:
Finally, remove boilerplate code from app.component.html and replace it with the demo component:
<app-demo></app-demo>
Now run the app. When values are submitted, they’re saved to our Firebase Cloudstore database, and the page will update in realtime as new records are added. Not bad for a few minutes of work!
At its core, a Trello board has just three pieces of functionality: create lists, create cards within the lists, and reorganize cards within and between the lists. The articles above give us all the tools we need to do this quickly ourselves, so let’s do it!
We’ll use the Angular CLI to do the following:
Create a new app
Add a list component
Add a board component
Make it pretty
Create New App
This first step’s pretty easy. We’re just going to use ng to create a new app and install the Angular CDK.
$ ng new lists-app --defaults
$ cd lists-app
$ npm install @angular/cdk
Do a quick check to make sure we’re starting in a good state:
$ ng serve --open
Add List Component
Now we’ll make our list component using ng generate component.
$ ng g c list
Hop into the code and make three changes. First, we need to import the DragDrop and Forms modules in src/app/app.module.ts:
import { BrowserModule } from '@angular/platform-browser';
import { DragDropModule } from '@angular/cdk/drag-drop';
import { FormsModule } from '@angular/forms';
import { NgModule } from '@angular/core';
import { AppComponent } from './app.component';
import { ListComponent } from './list/list.component';
@NgModule({
declarations: [
AppComponent,
ListComponent
],
imports: [
BrowserModule,
DragDropModule,
FormsModule
],
providers: [],
bootstrap: [AppComponent]
})
export class AppModule { }
Next, update src/app/list/list.component.ts to support drag/drop and dynamic creation of items. We’ll add the items array to store our list items, a drop function to handle our drop/drop event, and an onSubmit function to add new items to our list.
The third step is to change our markup in src/app/list/list.component.html. This is just two parts, displaying the drag-&-droppable list items and accepting input for new items.
<div cdkDropList [cdkDropListData]="items" (cdkDropListDropped)="drop($event)">
<div *ngFor="let item of items" cdkDrag>{{item}}</div>
</div>
<form #newItemForm="ngForm" (ngSubmit)="onSubmit(newItemForm)">
<input name="newItem" ngModel type="text" placeholder="Enter a new item"><button type="submit">Add Item</button>
</form>
That’s it for our list component, but let’s make one more change to app/app.component.html so we can test. Replace its entire contents with the following:
<app-list></app-list>
Now let’s do another check-in. We should be able to add items to our list and move them around via drag & drop.
$ ng serve --open
Add Board Component
Once again, we look to ng generate component to create our board component.
$ ng g c board
Just like the list component allows for dynamic creation of list items, we want our board component to allow dynamic creation of lists. So, let’s modify src/app/board/board.component.ts to support this:
import { Component, OnInit } from '@angular/core';
import { ListComponent } from '../list/list.component';
@Component({
selector: 'app-board',
templateUrl: './board.component.html',
styleUrls: ['./board.component.css']
})
export class BoardComponent implements OnInit {
lists: ListComponent[] = [];
constructor() { }
ngOnInit(): void {
}
addList() {
var newList = new ListComponent();
this.lists.push(newList);
}
}
And make the markup changes in src/app/board/board.component.html. One thing to note is the use of cdkDropListGroup. This makes all the lists connected and allows dragging & dropping between them.
<button (click)="addList()">Add List</button>
<div class="list-container" cdkDropListGroup>
<app-list *ngFor="let list of lists"></app-list>
</div>
We’ll also modify src/app/board/board.component.css so that lists will be added horizontally.
Finally, we’ll update app.component.html to use our board component instead of a single list:
<app-board></app-board>
Our board component is complete, so let’s do another check-in.
$ ng serve --open
Make It Pretty
At this point, the hard part’s done. We have our core functionality implemented. We can add lists, add items to the lists, and move the items around. Now let’s make it look nice!
Begin by installing Angular Material:
$ ng add @angular/material
Then import the MatCard, MatButton, and MatInput modules in src/app.module.ts:
import { BrowserModule } from '@angular/platform-browser';
import { DragDropModule } from '@angular/cdk/drag-drop';
import { FormsModule } from '@angular/forms';
import { MatButtonModule } from '@angular/material/button';
import { MatCardModule } from '@angular/material/card';
import { MatInputModule } from '@angular/material/input';
import { NgModule } from '@angular/core';
import { AppComponent } from './app.component';
import { ListComponent } from './list/list.component';
import { BoardComponent } from './board/board.component';
import { BrowserAnimationsModule } from '@angular/platform-browser/animations';
@NgModule({
declarations: [
AppComponent,
ListComponent,
BoardComponent
],
imports: [
BrowserModule,
DragDropModule,
FormsModule,
BrowserAnimationsModule,
MatButtonModule,
MatCardModule,
MatInputModule
],
providers: [],
bootstrap: [AppComponent]
})
export class AppModule { }
Now we can use mat-card, mat-raised-button, and mat-form-field in src/app/list/list.component.html:
The newest LTS version of Ubuntu is scheduled to release on April 23. Once it’s released, upgrading is as easy as running a few commands. In this article, we’ll walk through the process of updating the current system and then performing the upgrade.
Before you upgrade, make sure your system is as up to date as possible. Do this by running the follwing commands:
If you receive a No new release found message, it means the upgrade hasn’t been made available to you. It’s been noted that the upgrade path from 19.10 will not be enabled until a few days after release, and the upgrade from 18.04 LTS will not be enabled until a few days after the 20.04.1 release expected in late July. However, you can force the upgrade at your own risk by using the -d flag.
adamprescott.net 