Using Boolean Conditions to Filter Data in Python

Boolean filtering represents one of the most powerful data manipulation techniques in pandas. When you need to extract specific subsets of data based on conditions—such as retrieving all menu items under 500 calories—Boolean indexing provides an elegant and efficient solution. While our current dataset is limited, these filtering principles scale seamlessly to enterprise-level datasets containing millions of records.

Let's implement our first filter by creating a new DataFrame. This approach maintains data integrity by preserving the original dataset while generating focused views for analysis.

We'll create a new DataFrame called `max_500_cals_df` that equals our existing `food_df`, but with a crucial addition: a Boolean condition enclosed in square brackets. The syntax follows a clear pattern: `new_df = existing_df[boolean_condition]`. Our target column is calories, and our condition is straightforward: `calories <= 500`.

The filtering mechanism operates by evaluating each row against our Boolean condition. When we specify `food_df[food_df['calories'] <= 500]`, pandas iterates through every row, applies our condition to the calorie value, and accumulates only those rows returning `True` into the resulting DataFrame.

Executing this filter yields exactly two items: the pizza and the steak, both containing 500 calories or fewer. The hamburger and remaining steak entries exceed our threshold and are consequently excluded from the filtered result.

This row-by-row evaluation process demonstrates pandas' vectorized operations in action. Rather than writing explicit loops, pandas handles the iteration internally, applying your Boolean logic efficiently across the entire dataset. This approach becomes invaluable when working with datasets containing thousands or millions of records, where manual iteration would be prohibitively slow.

Now let's tackle a more complex filtering scenario. Your challenge: create a DataFrame containing only non-vegan food items. This requires targeting a different column type—Boolean rather than numeric—and applying inverse logic.

The solution involves creating a DataFrame called `non_vegan` and setting it equal to `food_df[food_df['vegan'] == False]`. This condition targets the vegan column and selects rows where the value equals `False`, effectively capturing three of our four items—everything except the garden salad.

The logic here is straightforward: if a food item's vegan status is `False`, we want to include it in our non-vegan collection. This demonstrates how Boolean filtering adapts to different data types while maintaining consistent syntax patterns.

Adding new data to existing DataFrames requires understanding pandas' location-based indexing system. The `df.loc[row_number]` method provides direct access to specific rows, and setting it equal to a list of values creates new entries seamlessly.

To add a new food item, we need four values corresponding to our existing columns: name (string), price (float), calories (integer), and vegan status (boolean). Let's define a new item: `new_item = ['Fruit Salad', 12.50, 180, True]`.

Our current DataFrame contains rows indexed 0 through 3. To add our fruit salad at the next available position, we use `food_df.loc[4] = new_item`. This command slots the new entry directly into index position 4, with each list value populating the corresponding column automatically.

The operation succeeds immediately, and our fruit salad appears at index 4, properly formatted and integrated into the existing data structure.

Here's your next challenge: add a bison burger to the DataFrame at the next available row position. You can either redefine the `new_item` variable or write the operation in a single line for maximum efficiency.

The solution requires incrementing our index to position 5: `food_df.loc[5] = ['Bison Burger', 18.50, 650, False]`. This demonstrates how manual index management works when you're certain about your DataFrame's current length.

---

Modifying existing data requires precise targeting of both row and column coordinates. To increase the bison burger's price by one dollar using the compound assignment operator, we need to specify both dimensions: `food_df.loc[5, 'price'] += 1`. This targets row 5 (our bison burger) and the price column specifically.

The `+=` operator provides a concise alternative to writing `food_df.loc[5, 'price'] = food_df.loc[5, 'price'] + 1`. When executed, the bison burger's price increases from $18.50 to $19.50, demonstrating precise cell-level data manipulation.

Bulk operations showcase pandas' true power for data transformation. To double all prices across the entire dataset, we use `food_df.loc[:, 'price'] *= 2`. The colon (`:`) represents all rows, while `'price'` specifies our target column. The `*= 2` operator multiplies every price by 2 simultaneously.

This vectorized operation affects every row instantly, demonstrating how pandas handles bulk transformations efficiently. Whether you're working with 10 rows or 10 million, the syntax and performance characteristics remain consistent.

To reverse this operation and restore original prices, we apply the inverse transformation: `food_df.loc[:, 'price'] /= 2`. Alternatively, `*= 0.5` achieves the same result, since multiplying by 0.5 equals dividing by 2. Choose the approach that best communicates your intent to future code readers.

Dynamic row addition eliminates the guesswork of manual index management. Instead of hardcoding index numbers, we can use `len(df)` to determine the next available position automatically. Since DataFrame indexing starts at 0, the length always equals the next available index.

For example, with 6 existing rows (indexed 0-5), `len(food_df)` returns 6—precisely the index we need for our new entry. Using `food_df.loc[len(food_df)] = new_item` ensures we always append to the end, regardless of the DataFrame's current size.

This dynamic approach proves invaluable in production environments where DataFrame lengths change frequently. Rather than tracking indices manually, let pandas calculate the appropriate position automatically.

Let's demonstrate with a Caesar salad: `food_df.loc[len(food_df)] = ['Caesar Salad', 14.75, 320, False]`. The salad appears at the correct position, and subsequent additions will automatically use the next available index.

Row removal requires careful consideration of your data structure goals. One approach involves slicing the DataFrame to exclude unwanted rows. If we accidentally created duplicate entries by running our addition command multiple times, we can remove excess rows using `food_df = food_df.loc[0:6, :]`.

This slice operation selects rows 0 through 6 (inclusive) and all columns, effectively removing any rows beyond index 6. Remember that `loc` includes the endpoint, so specifying `0:6` captures seven rows total (0, 1, 2, 3, 4, 5, 6).

When dealing with duplicate additions from repeated loop executions, adjust your slice accordingly. If you ran a three-item addition loop twice, use `food_df.loc[:-3, :]` to remove the last three rows, preserving the original additions while eliminating duplicates.

Automated bulk additions leverage loops to process multiple items efficiently. Consider a scenario where you need to add several menu items simultaneously. By bundling items into a parent list and iterating through them, we can automate the addition process.

Define your items as individual lists: `new_items = [['Chicken Salad', 16.25, 480, False], ['Chef Salad', 15.50, 350, False], ['Big Kahuna Burger', 22.00, 890, False]]`. Each sub-list contains the four required values for our DataFrame columns.

---

The loop structure iterates through each item, dynamically calculating the appropriate index: `for item in new_items: food_df.loc[len(food_df)] = item`. This approach scales efficiently, handling any number of new entries while maintaining proper indexing.

Each iteration recalculates `len(food_df)`, ensuring that as the DataFrame grows, new items are always added at the correct position. This dynamic length calculation prevents index conflicts and maintains data integrity throughout the bulk addition process.

Data modification targets specific cells using coordinate-based indexing. To change an existing entry—such as updating "Chef Salad" to "House Salad"—we need to identify both the row and column coordinates precisely.

Using `loc` with named column access: `food_df.loc[8, 'item'] = 'House Salad'` targets row 8 and the 'item' column specifically. This approach offers maximum clarity about which data element you're modifying, making your code self-documenting and maintainable.

For demonstration purposes, we can also use `iloc` (integer location) indexing: `food_df.iloc[8, 0] = 'Shrimp Salad'`. This targets row 8 and column 0 (the first column) using numeric coordinates. While more concise, `iloc` requires remembering column positions, making `loc` preferable for production code clarity.

Advanced filtering enables sophisticated data analysis scenarios. Challenge: extract all menu items priced at $15 or higher. This minimum price filter helps identify premium menu offerings and supports pricing strategy analysis.

The solution follows our established Boolean filtering pattern: `min_15_price_df = food_df[food_df['price'] >= 15]`. This condition evaluates each row's price column, accumulating only those items meeting our minimum threshold into the new DataFrame.

Such filtering operations become essential for business intelligence applications, where stakeholders need focused views of data subsets. Whether analyzing high-calorie items, premium-priced offerings, or vegan options, Boolean filtering provides the precision required for informed decision-making.

Statistical analysis transforms raw data into actionable insights. The `describe()` method generates comprehensive statistical summaries for all numeric columns in your DataFrame, providing eight key metrics that reveal data distribution patterns and central tendencies.

Execute `food_df.describe()` to generate a statistical overview covering count, mean, standard deviation, minimum, maximum, and three quartile values (25th, 50th, 75th percentiles). This analysis applies only to numeric columns—string and boolean columns are automatically excluded since statistical measures don't apply to categorical data.

Understanding standard deviation proves crucial for data analysis proficiency. If your mean calorie count is 637 with a standard deviation of 250, this indicates that approximately 68% of your food items fall between 387 calories (637 - 250) and 887 calories (637 + 250). This represents one standard deviation in either direction from the mean.

Expanding to two standard deviations captures approximately 95% of your data points, while three standard deviations encompass roughly 99.7%. These statistical principles, rooted in normal distribution theory, provide powerful frameworks for understanding data patterns and identifying outliers in your datasets.

Percentile interpretation offers another valuable analytical perspective. The 25th percentile indicates that 25% of items contain fewer calories than that threshold, while the 75th percentile means 75% of items fall below that value. The 50th percentile (median) often approximates the mean in well-distributed datasets, though small sample sizes may show significant variations.

These statistical insights become invaluable when scaling to enterprise datasets containing thousands of records. The `describe()` method provides instant overviews of data distribution, helping identify trends, outliers, and data quality issues that inform business decisions and analytical strategies.

---