9.7. Syntax Group

  • Catch expression results

  • Can be named or positional

  • Note, that for backreference, must use raw-sting or double backslash

  • () - matches whatever regular expression is inside the parentheses, and indicates the start and end of a group

  • (...) - unnamed group

  • (?P<mygroup>...) - named group mygroup

  • (?:...) - non-capturing group

  • (?#...) - comment

9.7.1. Positional Group

  • (...) - unnamed (positional) group

>>> import re
>>> TEXT = 'Mark Watney of Ares 3 landed on Mars on: Nov 7th, 2035 at 1:37 pm'
>>> re.findall(r'\dth', TEXT)
['7th']
>>>
>>> re.findall(r'(\d)th', TEXT)
['7']
>>>
>>> re.findall(r'\d(th)', TEXT)
['th']
>>> re.findall(r'\d:\d\d', TEXT)
['1:37']
>>> re.findall(r'(\d):\d\d', TEXT)
['1']
>>> re.findall(r'\d:(\d\d)', TEXT)
['37']
>>> re.findall(r'(\d):(\d\d)', TEXT)
[('1', '37')]
>>> re.findall(r'([A-Z][a-z]+\s[A-Z][a-z]+)', TEXT)
['Mark Watney']
>>>
>>> re.findall(r'([A-Z][a-z]+) ([A-Z][a-z]+)', TEXT)
[('Mark', 'Watney')]
>>>
>>> re.findall(r'([A-Z][a-z]+) ([A-Z][a-z]+)', TEXT)[0]
('Mark', 'Watney')
>>> firstname = r'([A-Z][a-z]+)'
>>> lastname = r'([A-Z][a-z]+)'
>>>
>>> re.findall(f'{firstname} {lastname}', TEXT)[0]
('Mark', 'Watney')
>>> firstname = r'([A-Z][a-z]+)'
>>> lastname = r'([A-Z][a-z]+)'
>>> name = f'{firstname} {lastname}'
>>>
>>> re.findall(name, TEXT)[0]
('Mark', 'Watney')
>>> firstname = r'[A-Z][a-z]+'
>>> lastname = r'[A-Z][a-z]+'
>>> name = f'({firstname}) ({lastname})'
>>>
>>> re.findall(name, TEXT)[0]
('Mark', 'Watney')

9.7.2. Named Group

  • (?P<mygroup>...) - named group mygroup

>>> import re
>>> TEXT = 'Mark Watney of Ares 3 landed on Mars on: Nov 7th, 2035 at 1:37 pm'
>>> firstname = r'[A-Z][a-z]+'
>>> lastname = r'[A-Z][a-z]+'
>>> name = f'(?P<firstname>{firstname}) (?P<lastname>{lastname})'
>>>
>>> re.findall(name, TEXT)
[('Mark', 'Watney')]
>>>
>>> re.search(name, TEXT)
<re.Match object; span=(0, 11), match='Mark Watney'>
>>>
>>> re.search(name, TEXT).groups()
('Mark', 'Watney')
>>>
>>> re.search(name, TEXT).groupdict()
{'firstname': 'Mark', 'lastname': 'Watney'}
>>> TEXT = 'Mark Watney of Ares 3 landed on Mars on: Nov 7th, 2035 at 1:37 pm'
>>> time = r'(?P<hour>\d{1,2}):(?P<minute>\d{1,2})'
>>>
>>> re.findall(time, TEXT)
[('1', '37')]
>>>
>>> re.search(time, TEXT).groups()
('1', '37')
>>>
>>> re.search(time, TEXT).group(0)
'1:37'
>>>
>>> re.search(time, TEXT).group(1)
'1'
>>>
>>> re.search(time, TEXT).group(2)
'37'
>>>
>>> re.search(time, TEXT).groupdict()
{'hour': '1', 'minute': '37'}

9.7.3. Non-Capturing Group

  • (?:...)

>>> import re
>>> TEXT = 'Mark Watney of Ares 3 landed on Mars on: Nov 7th, 2035 at 1:37 pm'
>>> re.findall(r'\w{3} \d{1,2}th, \d{4}', TEXT)
['Nov 7th, 2035']
>>>
>>> re.findall(r'\w{3} \d{1,2}st|nd|rd|th, \d{4}', TEXT)
['nd', 'th, 2035']
>>>
>>> re.findall(r'\w{3} \d{1,2}(st|nd|rd|th), \d{4}', TEXT)
['th']
>>>
>>> re.findall(r'\w{3} \d{1,2}(?:st|nd|rd|th), \d{4}', TEXT)
['Nov 7th, 2035']
>>>
>>> re.findall(r'(\w{3}) (\d{1,2})(?:st|nd|rd|th), (\d{4})', TEXT)
[('Nov', '7', '2035')]
>>>
>>> re.findall(r'(\w{3}) (\d{1,2})(st|nd|rd|th), (\d{4})', TEXT)
[('Nov', '7', 'th', '2035')]
>>> date = r'(\w{3} \d{1,2}(?:st|nd|rd|th), \d{4})'
>>> re.findall(date, TEXT)
['Nov 7th, 2035']
>>> year = r'\d{4}'
>>> month = r'\w{3}'
>>> day = r'\d{1,2}'
>>>
>>> re.findall(f'{month} {day}(st|nd|rd|th), {year}', TEXT)
['th']
>>>
>>> re.findall(f'{month} {day}(?:st|nd|rd|th), {year}', TEXT)
['Nov 7th, 2035']

9.7.4. Comment

  • (?#...) - comment

>>> import re
>>> TEXT = 'Mark Watney of Ares 3 landed on Mars on: Nov 7th, 2035 at 1:37 pm'
>>> re.findall(r'\d{4}(?#year)', TEXT)
['2035']
>>>
>>> re.findall(r'\d{1,2}(?#hour):\d{2}(?#minute)', TEXT)
['1:37']
>>> hour = r'\d{1,2}(?#hour)'
>>> minute = r'\d{2}(?#minute)'
>>> time = f'{hour}:{minute}'
>>>
>>> re.findall(time, TEXT)
['1:37']
>>>
>>> time
'\\d{1,2}(?#hour):\\d{2}(?#minute)'

9.7.5. Backreference

  • \g<number> - backreferencing by group number

  • \g<name> - backreferencing by group name

  • (?P=name) - backreferencing by group name

  • \number - backreferencing by group number

>>> year = r'(?P<year>\d{4})'
>>> month = r'(?P<month>[A-Z][a-z]{2})'
>>> day = r'(?P<day>\d{1,2})'
>>> date = f'{month} {day}(?:st|nd|rd|th), {year}'
>>>
>>> TEXT = 'Mark Watney of Ares 3 landed on Mars on: Nov 7th, 2035 at 1:37 pm'
>>>
>>>
>>> re.sub(date, '\g<3> \g<1> \g<2>', TEXT)
'Mark Watney of Ares 3 landed on Mars on: 2035 Nov 7 at 1:37 pm'
>>>
>>> re.sub(date, '\g<year> \g<month> \g<day>', TEXT)
'Mark Watney of Ares 3 landed on Mars on: 2035 Nov 7 at 1:37 pm'

Although this is not working in Python:

>>> re.sub(f'{month} {day}th, {year}', '(?P=day) (?P=month) (?P=year)', TEXT)
'Mark Watney of Ares 3 landed on Mars on: (?P=day) (?P=month) (?P=year) at 1:37 pm'

Example:

>>> html = '<p>We choose to go to the <strong>Moon</strong></p>'
>>>
>>>
>>> re.findall('<(?P<tagname>[a-z]+)>.*</(?P=tagname)>', html)
['p']
>>>
>>> re.findall('<(?P<tagname>[a-z]+)>(.*)</(?P=tagname)>', html)
[('p', 'We choose to go to the <strong>Moon</strong>')]

9.7.6. Examples

  • (\w+) - word character (including unicode chars, numbers an underscores)

  • \d+(\.\d+)? - float with optional decimals

  • \d+(,\d+)? - number with coma (,) as thousands separator

  • (?P<word>\w+) - name group word with \w+ with at least one word character (including unicode chars, numbers an underscores)

  • (?P<tag><.*?>).+(?P=tag) - matches text inside of a <tag> (opening and closing tag is the same)

  • (.+) \1 - matches the the or 55 55

  • (.+) \1 - not matches thethe (note the space after the group)

>>> import re
>>> TEXT = 'Mark Watney of Ares 3 landed on Mars on: Nov 7th, 2035 at 1:37 pm'
>>> re.findall(r'\d{,2}(st|nd|rd|th)?', TEXT)  
['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '',
 '', '', '', '', '', 'nd', '', '', '', '', '', '', '', '', '', '', '', '',
 '', '', '', '', '', '', '', 'th', '', '', '', '', '', '', '', '', '', '',
 '', '', '', '', '']
>>>
>>> re.findall(r'\d{1,2}(st|nd|rd|th)?', TEXT)
['', 'th', '', '', '', '']
>>>
>>> re.findall(r'\d{1,2}(st|nd|rd|th)+?', TEXT)
['th']
>>>
>>> re.findall(r'\d{1,2}st|nd|rd|th+?', TEXT)  # nd is also in word `landed`
['nd', 'th']
>>>
>>> re.findall(r'\d{1,2}(?:st|nd|rd|th)+?', TEXT)
['7th']
>>>
>>> re.findall(r'(\d{1,2})(st|nd|rd|th)+?', TEXT)
[('7', 'th')]
>>>
>>> re.findall(r'(\d{1,2})(?:st|nd|rd|th)+?', TEXT)
['7']
>>>
>>> re.findall(r'(\w{3}) (\d{1,2})(?:st|nd|rd|th)+?, (\d{4})', TEXT)
[('Nov', '7', '2035')]
>>>
>>> re.findall(r'(\w{3}) (\d{1,2})(?:st|nd|rd|th)+?, (\d{4})', TEXT)[0]
('Nov', '7', '2035')
>>>
>>> re.findall(r'(\w{3} \d{1,2}(?:st|nd|rd|th)+?, \d{4})', TEXT)
['Nov 7th, 2035']

9.7.7. Use Case - 0x01

  • Dates

>>> import re
>>> TEXT = 'Mark Watney of Ares 3 landed on Mars on: Nov 7th, 2035 at 1:37 pm'
>>> year = r'(?P<year>\d{4})'
>>> month = r'(?P<month>\w{3})'
>>> day = r'(?P<day>\d{1,2}(?:st|nd|rd|th)+?)'
>>> date = f'{month} {day}, {year}'
>>>
>>> re.search(date, TEXT).groupdict()
{'month': 'Nov', 'day': '7th', 'year': '2035'}

9.7.8. Use Case - 0x02

>>> import re
>>> line = 'value=123'
>>>
>>> re.findall(r'(\w+)\s?=\s?(\d+)', line)
[('value', '123')]
>>> line = 'value = 123'
>>>
>>> re.findall(r'(\w+)\s?=\s?(\d+)', line)
[('value', '123')]

9.7.9. Use Case - 0x03

>>> import re
>>>
>>>
>>> variable = r'(?P<variable>\w+)'
>>> space = r'\s?'  # optional space
>>> value = r'(?P<value>.+)'
>>> assignment = f'^{variable}{space}={space}{value}$'
>>>
>>> line_of_code = 'myvar = 123'
>>> re.findall(assignment, line_of_code)
[('myvar', '123')]

9.7.10. Use Case - 0x04

>>> import re
>>>
>>>
>>> variable = r'(?P<variable>\w+)'
>>> space = r'\s?(?#optional space)'
>>> value = r'(?P<value>.+)'
>>> assignment = f'^{variable}{space}={space}{value}$'
>>>
>>> assignment
'^(?P<variable>\\w+)\\s?(?#optional space)=\\s?(?#optional space)(?P<value>.+)$'

9.7.11. Use Case - 0x05

>>> import re
>>>
>>>
>>> HTML = '<p>Hello World</p>'
>>>
>>> search = r'<p>(.+)</p>'
>>> replace = r'<strong>\g<1></strong>'
>>>
>>> re.sub(search, replace, HTML)
'<strong>Hello World</strong>'

9.7.12. Use Case - 0x06

>>> import re
>>>
>>>
>>> HTML = '<p>Hello World</p>'
>>>
>>> search = r'<p>(?P<text>.+)</p>'
>>> replace = r'<strong>\g<text></strong>'
>>>
>>> re.sub(search, replace, HTML)
'<strong>Hello World</strong>'

9.7.13. Use Case - 0x07

>>> import re
>>>
>>>
>>> HTML = '<p>Hello World</p>'
>>> tag = re.findall(r'<(?P<tag>.+)>(?:.+)</(?P=tag)>', HTML)
>>>
>>> tag
['p']

9.7.14. Use Case - 0x08

>>> import re
>>>
>>>
>>> HTML = '<p>Hello World</p>'
>>>
>>> re.findall(r'<(?P<tag>.*?)>(.*?)</(?P=tag)>', HTML)
[('p', 'Hello World')]

9.7.15. Assignments

Code 9.24. Solution
"""
* Assignment: RE Syntax Group
* Complexity: easy
* Lines of code: 1 lines
* Time: 3 min

English:
    1. Use regular expressions find in text:
        a. all dates (month name followed by day number)
    2. Run doctests - all must succeed

Polish:
    1. Użyj wyrażeń regularnych wyszukiwania w tekście:
        a. wszyskie daty (miesiąc po którym jest dzień)
    2. Uruchom doctesty - wszystkie muszą się powieść

Hint:
    * `re.findall()`
    * After day there is an ordinal: st, nd, rd, th (you can use: ..)

References:
    [1] Authors: Wikipedia contributors
        Title: Apollo 11
        Publisher: Wikipedia
        Year: 2019
        Retrieved: 2019-12-14
        URL: https://en.wikipedia.org/wiki/Apollo_11

Tests:
    >>> import sys; sys.tracebacklimit = 0
    >>> from pprint import pprint

    >>> pprint(result, compact=True)
    ['July 20', 'July 21']
"""

import re

TEXT = """Apollo 11 was the American spaceflight that first landed
humans on the Moon. Commander (CDR) Neil Armstrong and lunar module
pilot (LMP) Buzz Aldrin landed the Apollo Lunar Module (LM) Eagle on
July 20th, 1969 at 20:17 UTC, and Armstrong became the first person
to step (EVA) onto the Moon's surface (EVA) 6 hours 39 minutes later,
on July 21st, 1969 at 02:56:15 UTC. Aldrin joined him 19 minutes later.
They spent 2 hours 31 minutes exploring the site they had named Tranquility
Base upon landing. Armstrong and Aldrin collected 47.5 pounds (21.5 kg)
of lunar material to bring back to Earth as pilot Michael Collins (CMP)
flew the Command Module (CM) Columbia in lunar orbit, and were on the
Moon's surface for 21 hours 36 minutes before lifting off to rejoin
Columbia."""

# Find all dates (month name followed by day number)
# Note: after day there is an ordinal: st, nd, rd, th (you can use: ..)
# Example: 'July 20', 'July 21'
# type: list[str]
result = ...

Code 9.25. Solution
"""
* Assignment: RE Syntax Group
* Complexity: easy
* Lines of code: 2 lines
* Time: 3 min

English:
    1. Use regular expressions find in text:
        a. durations in list[tuple] format
           example: 6 hours 39 minutes -> [('6', '39'), ...]
    2. Run doctests - all must succeed

Polish:
    1. Użyj wyrażeń regularnych wyszukiwania w tekście:
        a. okresy w formacie list[tuple]
           przykład: 6 hours 39 minutes -> [('6', '39'), ...]
    2. Uruchom doctesty - wszystkie muszą się powieść

Hint:
    * `re.findall()`

References:
    [1] Authors: Wikipedia contributors
        Title: Apollo 11
        Publisher: Wikipedia
        Year: 2019
        Retrieved: 2019-12-14
        URL: https://en.wikipedia.org/wiki/Apollo_11

Tests:
    >>> import sys; sys.tracebacklimit = 0
    >>> from pprint import pprint

    >>> pprint(result)
    [('6', '39'), ('2', '31'), ('21', '36')]
"""

import re

TEXT = """Apollo 11 was the American spaceflight that first landed
humans on the Moon. Commander (CDR) Neil Armstrong and lunar module
pilot (LMP) Buzz Aldrin landed the Apollo Lunar Module (LM) Eagle on
July 20th, 1969 at 20:17 UTC, and Armstrong became the first person
to step (EVA) onto the Moon's surface (EVA) 6 hours 39 minutes later,
on July 21st, 1969 at 02:56:15 UTC. Aldrin joined him 19 minutes later.
They spent 2 hours 31 minutes exploring the site they had named Tranquility
Base upon landing. Armstrong and Aldrin collected 47.5 pounds (21.5 kg)
of lunar material to bring back to Earth as pilot Michael Collins (CMP)
flew the Command Module (CM) Columbia in lunar orbit, and were on the
Moon's surface for 21 hours 36 minutes before lifting off to rejoin
Columbia."""

# Find durations from text in list[tuple] format
# Example: [('6', '39'), ('2', '31'), ('21', '36')]
# type: list[str]
result = ...

Code 9.26. Solution
"""
* Assignment: RE Syntax Group
* Complexity: easy
* Lines of code: 2 lines
* Time: 3 min

English:
    1. Use regular expressions find in text:
        a. durations in list[dict] format
           example: 6 hours 39 minutes -> [{'hours': '6', 'minutes': '39'}, ...]
    2. Run doctests - all must succeed

Polish:
    1. Użyj wyrażeń regularnych wyszukiwania w tekście:
        a. okresy w formacie list[dict]
           przykład: 6 hours 39 minutes -> [{'hours': '6', 'minutes': '39'}, ...]
    2. Uruchom doctesty - wszystkie muszą się powieść

Hint:
    * `re.findall()`

References:
    [1] Authors: Wikipedia contributors
        Title: Apollo 11
        Publisher: Wikipedia
        Year: 2019
        Retrieved: 2019-12-14
        URL: https://en.wikipedia.org/wiki/Apollo_11

Tests:
    >>> import sys; sys.tracebacklimit = 0
    >>> from pprint import pprint

    >>> pprint(result)
    [{'hours': '6', 'minutes': '39'},
     {'hours': '2', 'minutes': '31'},
     {'hours': '21', 'minutes': '36'}]
"""

import re

TEXT = """Apollo 11 was the American spaceflight that first landed
humans on the Moon. Commander (CDR) Neil Armstrong and lunar module
pilot (LMP) Buzz Aldrin landed the Apollo Lunar Module (LM) Eagle on
July 20th, 1969 at 20:17 UTC, and Armstrong became the first person
to step (EVA) onto the Moon's surface (EVA) 6 hours 39 minutes later,
on July 21st, 1969 at 02:56:15 UTC. Aldrin joined him 19 minutes later.
They spent 2 hours 31 minutes exploring the site they had named Tranquility
Base upon landing. Armstrong and Aldrin collected 47.5 pounds (21.5 kg)
of lunar material to bring back to Earth as pilot Michael Collins (CMP)
flew the Command Module (CM) Columbia in lunar orbit, and were on the
Moon's surface for 21 hours 36 minutes before lifting off to rejoin
Columbia."""

# Extract duration values from text in list[dict] format
# Example: [{'hours': '6', 'minutes': '39'}, {'hours': '2', 'minutes': '31'}, ...]
# type: list[str]
result = ...